Smart necklace with stereo vision and onboard processing

ABSTRACT

A method for providing directions to a blind user of a smart device is described. The method includes detecting, by at least two sensors and in response to a selection of a find mode of the smart device, image data corresponding to a surrounding environment of the smart device and positioning data corresponding to a positioning of the smart device. The method also includes receiving, by an input device, the desired object or the desired location. The method also includes determining, by a processor, the initial location of the smart device based on the image data, the positioning data and map data stored in a memory of the smart device. The method also includes providing, by the output device, the directions to the desired object based on the initial location of the smart device and the map data.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.14/480,575, filed on Sep. 8, 2014, which is a continuation-in-part ofU.S. application Ser. No. 14/154,714, filed on Jan. 14, 2014, the entirecontents of both applications are hereby incorporated by referenceherein.

BACKGROUND

1. Field

The present disclosure relates to a wearable device which provideshaptic and audio feedback based on stereo camera input.

2. Description of the Related Art

Wearable cameras provide recording and documenting of a user'sexperience, often from the same or similar point of view or field ofview (FOV) of the user. However, these devices are passive recorders,and do not generally provide real time processing and information aboutthe scene in the FOV. Certain users, such as blind persons, may desireadditional feedback relating to the environment. Other wearable camerasmay be designed to assist blind persons. However, such devices lackstereo cameras for reliable depth perception information.

Thus, there is a need for an unobtrusive device which augments a user'senvironmental awareness and social interaction with depth perception andobject recognition.

SUMMARY

What is described is a method for providing directions to a blind userof a smart device. The directions are from an initial location of thesmart device to a location of a desired object or a desired location.The method includes detecting, by at least two sensors and in responseto a selection of a find mode of the smart device, image datacorresponding to a surrounding environment of the smart device andpositioning data corresponding to a positioning of the smart device. Themethod also includes receiving, by an input device, the desired objector the desired location. The method also includes determining, by aprocessor, the initial location of the smart device based on the imagedata, the positioning data and map data stored in a memory of the smartdevice. The method also includes providing, by the output device, thedirections to the desired object based on the initial location of thesmart device and the map data.

Also described is a method for describing at least one object or personwithin a predetermined distance and angle of a smart device to a blinduser of the smart device. The method includes storing, in a memory,stored image data associated with a plurality of stored objects andpeople and a plurality of identifiers such that each identifier isassociated with a stored object or person. The method also includesdetecting, by a camera and in response to a selection of an explore modeor a scan mode of the smart device, detected image data corresponding tothe at least one object or person. The method also includes determining,by a processor, at least a first identifier from the plurality ofidentifiers corresponding to the at least one object or person withinthe predetermined distance and angle of the camera based on the detectedimage data and the stored image data. The method also includesoutputting, via a speaker, the at least first identifier.

Also described is a method for storing a first location of a smartdevice and providing directions to a blind user from a second locationof the smart device to the first location of the smart device. Themethod includes detecting, by a positioning sensor, a first positioningdata corresponding to a first location of the smart device and a secondpositioning data corresponding to a second location of the smart device.The method also includes storing, in a memory and in response to a firstselection of the capture mode of the smart device, the first positioningdata on a map. The method also includes determining, by a processor andin response to a second selection of a capture mode of the smart device,directions to the first location from the second location based on thefirst positioning data and the second positioning data. The method alsoincludes providing, by an output device, the directions.

BRIEF DESCRIPTION OF THE DRAWINGS

Other systems, methods, features, and advantages of the presentinvention will be or will become apparent to one of ordinary skill inthe art upon examination of the following figures and detaileddescription. It is intended that all such additional systems, methods,features, and advantages be included within this description, be withinthe scope of the present invention, and be protected by the accompanyingclaims. Component parts shown in the drawings are not necessarily toscale, and may be exaggerated to better illustrate the importantfeatures of the present invention. In the drawings, like referencenumerals designate like parts throughout the different views, wherein:

FIG. 1A is a block diagram of a smart necklace according to anembodiment of the present invention;

FIG. 1B illustrates modules within a processor of the smart necklace ofFIG. 1A according to an embodiment of the present invention;

FIG. 2 illustrates an embodiment of a smart necklace as viewed from afront of the smart necklace according to an embodiment of the presentinvention;

FIG. 3 illustrates the smart necklace of FIG. 2 from a back of the smartnecklace according to an embodiment of the present invention;

FIG. 4 illustrates a portion of the smart necklace of FIG. 2 includingpart of an upper portion 201, a right middle portion and a lower rightportion according to an embodiment of the present invention;

FIG. 5 illustrates a portion of the smart necklace of FIG. 2 includingpart of an upper portion, a left middle portion and a lower left portionaccording to an embodiment of the present invention;

FIG. 6 illustrates a smart necklace according to an embodiment of thepresent invention;

FIG. 7 illustrates a view of the smart necklace of FIG. 6 from a bottomof the smart necklace according to an embodiment of the presentinvention;

FIG. 8A illustrates a portable charging unit for use with a smartnecklace according to an embodiment of the present invention;

FIG. 8B illustrates the portable charging unit of FIG. 8A connected to asmart necklace according to an embodiment of the present invention;

FIG. 9A illustrates a strap attachment that is adapted to attach distalends of a smart necklace to each other according to an embodiment of thepresent invention;

FIG. 9B illustrates the strap attachment of FIG. 9A connected to a smartnecklace according to an embodiment of the present invention;

FIG. 10A illustrates a first insert that is configured to be attached toa smart necklace in order to enlarge the size of the smart necklaceaccording to an embodiment of the present invention;

FIG. 10B illustrates a second insert that is configured to be attachedto a smart necklace in order to enlarge the size of the smart necklaceaccording to an embodiment of the present invention;

FIG. 10C illustrates a smart necklace including the first insert of FIG.10A and where a upper portion of the smart necklace is disconnected froma left middle portion of the smart necklace according to an embodimentof the present invention;

FIG. 11A illustrates a charging unit configured to charge a smartnecklace according to an embodiment of the present invention;

FIG. 11B illustrates a smart necklace positioned on the charging unit ofFIG. 11A according to an embodiment of the present invention;

FIG. 12 illustrates a method for updating location information in amemory of a smart device according to an embodiment of the presentinvention;

FIG. 13 illustrates a method for selection of a mode of a smart deviceaccording to an embodiment of the present invention;

FIG. 14 illustrates a method to be performed if a selected mode of asmart device is a find mode according to an embodiment of the presentinvention;

FIG. 15 illustrates an exemplary implementation of the method of FIG. 14according to an embodiment of the present invention;

FIG. 16 illustrates a method to be performed if a selected mode of asmart device is an explore mode according to an embodiment of thepresent invention;

FIG. 17 illustrates an exemplary implementation of the method of FIG. 16according to an embodiment of the present invention;

FIG. 18 illustrates a method to be performed if a selected mode of asmart device is a scan mode according to an embodiment of the presentinvention;

FIG. 19 illustrates an exemplary implementation of the method of FIG. 18according to an embodiment of the present invention;

FIG. 20 illustrates a method to be performed if a selected mode of asmart device is a capture mode according to an embodiment of the presentinvention;

FIG. 21 illustrates an exemplary implementation of the method of FIG. 20according to an embodiment of the present invention; and

FIG. 22 illustrates a graphical user interface displayed on a display ofa smart device that illustrates various settings of the smart deviceaccording to an embodiment of the present invention.

DETAILED DESCRIPTION

Blind people may be at a disadvantage because of a lack of ability tosense their environment through their vision. Described herein is asmart necklace that is to be worn around a neck of a user and is capableof providing environmental and social awareness to a user. The smartnecklace can detect image and location information within the user'senvironment and use this image information to assist the user in theirdaily life.

The smart necklace is particularly well suited to assist blind orvisually impaired users in their daily lives. The smart necklace isadapted to provide a variety of information based on requests from theuser and detected environmental data. For example, the smart necklacecan identify objects within a particular field of view (FOV) as well asprovide depth information regarding the objects to the user, providenavigation instructions to the user to indoor and outdoor locations aswell as learn labels for new objects, people and places so that thesmart necklace can later identify the labeled objects, people andplaces.

The use of stereo cameras within a device such as the smart necklace isadvantageous over the current state of the art. The use of stereocameras allows depth information to be detected. This depth informationcan be extremely valuable, especially to a blind user who has limiteddistance detection capabilities. Similarly, the combinations of inputsand outputs provide benefits not currently available. The selection ofinputs, including cameras that are always detecting image data, a GPSthat collects location data and an IMU that collects acceleration data,allows the smart necklace to navigate the user within an enclosed areawhere location data alone may not be helpful. Similarly, the combinationof stereo audio output and stereo haptic output ensures that the usercan always receive output data from the smart necklace in an easy tocomprehend manner.

This disclosure further discloses methods to be performed by a wearabledevice for assisting blind or visually impaired users. The methods aredesigned to determine data to provide to the user based on data from apair of stereo cameras, an IMU and a GPS. The methods may help identifyobjects in the user's surroundings, navigate the user to a particularlocation and learn new people, places and things based on input from theuser.

The first method is adapted to direct a user to a location of an object,person or place. The second method and the third method are adapted todescribe objects within a predetermined area of the smart device. Thefourth method is adapted to capture a present location of the smartdevice, store the location and then direct the user to the storedlocation from a second location. These methods are advantageous over thecurrent state of the art because they allow data to be detected andprovided to the user based on a combination of positioning data andimage data. In particular, the use of an IMU and cameras in determininglocation information provides more accurate positioning than atraditional GPS based device.

In one implementation, a smart necklace 100 (or blind aid necklace)includes an onboard processing array 110, which communicates with asensor array 120, an interface array 130, and a component array 140. Theonboard processing array 110, the sensor array 120, the interface array130, and the component array 140 are exemplary groupings to visuallyorganize the components of the smart necklace 100 in the block diagramof FIG. 1A and are not limiting or necessarily representative of anyphysical groupings. In addition, certain implementations may have moreor less components than illustrated in FIG. 1A.

The onboard processing array 110 includes a processor 111 and a memory112. The processor 111 may be a computer processor such as an ARMprocessor, DSP processor, distributed processor, or other form ofcentral processing. The processor 111 may be positioned on the smartnecklace 100, may be a remote processor or it may be a pairing of alocal and a remote processor.

The memory 112 may be one or any combination of the following: a RAM orother volatile or nonvolatile memory, a non-transitory memory or a datastorage device, such as a hard disk drive, a solid state disk drive, ahybrid disk drive, or other appropriate data storage, and may furtherstore machine-readable instructions, which may be loaded into the memory112 and executed by the processor 111. As with the processor 111, thememory 112 may be positioned on the smart necklace 100, may bepositioned remote from the smart necklace 100 or it may be a pairing ofa local and a remote memory.

The sensor array 120 includes stereo cameras 121, a camera 122, aninertial measurement unit (IMU) 123, a global positioning system (GPS)124, and a sensor 125. The stereo cameras 121 may be a stereo camerapair comprising two cameras offset by a stereo distance. The stereodistance may be optimized for the two cameras. The smart necklace 100may have more than one pair of stereo cameras 121. The camera 122 may bea camera or other optical sensor not part of a stereo camera pair. TheIMU 123 may be an IMU which may further comprise one or more of anaccelerometer, a gyroscope, a magnetometer or the like. The GPS 124 maybe one or more GPS units. The sensor 125 may be one or more sensorswhich provide further information about the environment in conjunctionwith the rest of the sensor array 120. The sensor 125 may be one or moreof a camera, a temperature sensor, an air pressure sensor, a moisture orhumidity sensor, a gas detector or other chemical sensor, a soundsensor, a pH sensor, a smoke detector, a metal detector, an actinometer,an altimeter, a depth gauge, a compass, a radiation sensor, a motiondetector, a light sensor or other sensor.

The interface array 130 includes a microphone 131, a speaker 132, avibration unit 133, an input device 134, and a display 135. Themicrophone 131 may be a microphone or other device capable of receivingsounds, such as voice activation/commands or other voice actions fromthe user, and may be integrated with or external to the smart necklace100. The speaker 132 may be one or more speakers or other devicescapable of producing sounds and/or vibrations. The vibration unit 133may be a vibration motor or actuator capable of providing haptic andtactile output. In certain implementations, the vibration unit 133 mayalso be capable of producing sounds, such that the speaker 132 and thevibration unit 133 may be the same or integrated. The vibration unit 133may include a left vibration motor in the left portion, and a rightvibration motor in the right portion. This advantageously allows variouscombinations of haptic feedback using a left-side vibration that maydiffer from a right-side vibration.

The input device 134 may be an input device such as a touch sensorand/or one or more buttons. For example, the input device 134 may be aplurality of buttons, such that each button corresponds to a differentactivity of the smart necklace 100. In various embodiments, themicrophone 131 may be considered an input device, such that the term“input device” may refer to the microphone, a button or buttons, atouchpad, a touchscreen or the like.

The display 135 may be a display, integrated into the smart necklace 100or wirelessly connected to the smart necklace 100, and may be capable ofdisplaying visual data from the stereo cameras 121 and/or the camera122. In other implementations, the display 135 may be another visualalert device, such as one or more LEDs or similar light source. Invarious embodiments, the input device 134 and the display 135 may be thesame or integrated, such as a touchscreen.

The component array 140 includes a battery 141, an antenna 142, and aninput/output port (I/O port) 143. The battery 141 may be a battery orother power supply capable of powering the smart necklace 100. Thebattery 141 may have a connection port for recharging, or may bewirelessly recharged, such as through induction charging. The antenna142 may be one or more antennas capable of transmitting and receivingwireless communications. For example, the antenna 142 may be a Bluetoothor WiFi antenna, may be a radio frequency identification (RFID) antennaor reader, and/or a near field communication (NFC) unit. The I/O port143 may be one or more ports for connecting additional peripherals. Forexample, the I/O port 143 may be a headphone jack, or may be a dataport.

The antenna 142 and/or the I/O port 143 allows the smart necklace 100 toconnect to another device or network for data downloads, such as updatesto the smart necklace, map information or other relevant information fora particular application, and data uploads, such as status updates andupdated map information. Further, the antenna 142 and/or the I/O port143 allows the smart necklace 100 to communicate with other smartdevices for distributed computing or sharing resources.

The smart necklace 100 described herein is generally a stand-alonedevice. However, in other implementations, the smart necklace 100 may beconfigured or optimized to work in conjunction with other devices. Forexample, smartphones, tablets, or other mobile devices may wirelesslyconnect to the smart necklace 100 for shared resources and processing.The mobile device may act as a display unit for the smart necklace 100.The smart necklace 100 may further have specific protocols forinteracting with mobile devices or other smart necklaces. Additionally,the smart necklace 100 may connect over the interne to remote processingand/or remote storage, such as a cloud.

The smart necklace 100 is a lightweight, wearable smart device that isworn around the user's neck for environmental awareness, navigation,social interactions, and obstacle avoidance through real-time feedback.The smart necklace 100 is capable of recognizing objects around theuser, in order to alert the user. For example, the smart necklace 100may be used by a blind person to aid in environmental awareness andnavigate safely around obstacles. The smart necklace 100 provides theuser audio and haptic feedback through the speaker 132 and/or thevibration unit 133 based upon input, such as camera input from thestereo cameras 121 and the camera 122.

Stereo cameras provide depth information in both indoor and outdoorenvironments. The stereo cameras 121 may face forward, in front of auser, to establish a field of view (FOV). The stereo cameras 121 mayhave, for example, an FOV between around 90 degrees and around 130degrees. The stereo cameras 121 provide 3D information such as depth infront of the user. Stereo cameras 121 having different focal lengths maybe provided. In various embodiments, one set of stereo cameras 121 maybe positioned a relatively small distance apart (such as on a singleside of the smart necklace 100) and another pair of stereo cameras 121may be positioned a relatively large distance apart (such as positioningone of the pair of stereo cameras 121 on a first side of the smartnecklace 100 and the other pair of stereo cameras 121 on a second sideof the smart necklace 100). The pair of stereo cameras 121 having therelatively small distance apart can have a relatively short focallength, and thus provide detailed depth information for objects that arerelatively close to the user. The pair of stereo cameras 121 having therelatively large distance apart can have a relatively long focal length,and thus provide detailed depth information for objects that arerelatively far away from the user.

One or more additional camera 122 may be placed to the sides of thestereo cameras 121 or on an opposite side of the smart necklace 100 fromthe pair of stereo cameras 121. Camera 122 may have a field of viewbetween 90 degrees and 130 degrees. In various embodiments, the cameras122 may be placed where needed, such as behind the user's neck toprovide data for an area behind the user.

Although the one or more camera 122 may be monocular, it can providesimple recognition, even without depth or distance information. Forexample, the cameras 122 can detect moving objects in the user'speriphery. The pair of stereo cameras 121 and the camera 122 maycontinuously passively recognize objects in the environment. The smartnecklace 100 may compare the image data of the object to image datastored in a local or cloud-based memory in order to identify the object.Working in conjunction with the other sensors in the sensor array 120,the smart necklace 100 can provide the user with guidance and navigationcommands by way of audio and haptic feedback. In a preferred embodiment,the pair of stereo cameras 121 are utilized in part because they canadvantageously provide depth information. In another embodiment, one ormore omnidirectional cameras may be utilized in addition to or in lieuof the pair of stereo cameras 12 and the camera 122.

The GPS 124 provides location information, which works with the inertialguidance information, including velocity and orientation information,provided by the IMU 123 to help direct the user. The GPS 124 and/or theIMU 123 may be considered a positioning sensor, as either device maydetect positioning information. The memory 112 may store, for example,map information or data to help locate and provide navigation commandsto the user. The map data may be preloaded, downloaded wirelesslythrough the antenna 142, or may be visually determined, such as bycapturing a building map posted near a building's entrance, or builtfrom previous encounters and recordings. The map data may be abstract,such as a network diagram with edges, or a series of coordinates withfeatures. The map data may contain points of interest to the user, andas the user walks, the stereo cameras 121 and/or cameras 122 maypassively recognize additional points of interest and update the mapdata.

For example, the user may give a voice command, “Take me to building Xin Y campus.” The smart necklace 100 may then download a relevant map ifnot already stored, or may navigate based on perceived images from thepair of stereo cameras 121 and the camera 122. As the user follows thenavigation commands from the smart necklace 100, the user may walk by acoffee shop in the morning, and the smart necklace 100 would recognizethe coffee shop and the time of day, along with the user's habits, andappropriately alert the user. The smart necklace 100 may verbally alertthe user through the speaker 132. The user may use the input device 134to adjust settings, which for example may control the types of alerts,what details to announce, and other parameters which may relate toobject recognition or alert settings. The user may turn on or offcertain features as needed.

When navigating indoors, the standalone GPS units may not provide enoughinformation to a blind user to navigate around obstacles and reachdesired locations or features. The smart necklace 100 may recognize, forinstance, stairs, exits, and restrooms and appropriately store them inthe memory 112. In another example, the smart necklace 100 may determineempty seats for the user to navigate to, or may remember the user'sspecific seat in order to navigate away and subsequently return to thesame seat. Other points of interest may be potential hazards,descriptions of surrounding structures, alternate routes, and otherlocations. Additional data and points of interest can be downloadedand/or uploaded to mobile devices and other devices, social networks, orthe cloud, through Bluetooth or other wireless networks. With wirelessconnectivity, local processing can be reduced, as high level data andprocessing may be available from the cloud or other remote data centers.

The smart necklace 100 may determine paths for navigation, which may befurther modified for the user's needs. For example, a blind person mayprefer routes that follow walls. Using the IMU 123 and/or the GPS 124and other sensors, the smart necklace 100 can determine the user'slocation and orientation to guide them along the path, avoidingobstacles. The vibration unit 133 and the speaker 132 provide audio andhaptic cues to help guide the user along the path.

For example, the speaker 132 may play a command to move forward aspecified distance. Then, special audio tones or audio patterns can playwhen the user is at a waypoint, and guide the user to make a turnthrough additional tones or audio patterns. A first tone, audio patternor vibration can alert the user to the start of a turn, such as a singletone or a vibration from the left side of the smart necklace mayindicate a left turn. A second tone, audio pattern or vibration canalert the user that the turn is complete such as two tones, or thevibration may stop, such as the left side ceases to vibrate when theturn is complete. Different tones, patterns or vibrations may alsosignify different degrees of turns, such as a specific tone for a 45degree turn and a specific tone for a 90 degree turn. Alternatively orin addition to tones and vibrations, the smart necklace 100 may provideverbal cues, similar to a car GPS navigation command.

High level alerts may also be provided through audio feedback. Forexample, as the smart necklace 100 reaches a predetermined distance—suchas a foot or other value which may be stored in the memory 112 and maybe adjusted—from an obstacle or hazard, the speaker 132 and/or thevibration unit 133 may provide audible alerts. As the smart necklace 100gets closer to the obstacle, the audible alerts may increase inintensity or frequency.

The vibration unit 133 may include a left vibration motor in the leftportion of the smart necklace 100 and a right vibration motor in theright portion of the smart necklace 100 for providing stereo hapticfeedback to the user. Vibration patterns on the left portion can bedifferent than vibration patterns on the right portion. In this manner,different combination of left/right vibration patterns can convey morevariety of useful information to the user (as opposed to outputting thesame pattern in both left and right vibration). For example, certainvibration patterns on the left that are lacking on the right may be usedto signal to the user that the user should turn left.

The microphone 131 may detect additional environmental data, such assounds of moving cars or other possible hazards. The microphone 131 maywork in conjunction with the speaker 132, and may be placed away fromthe speaker 132 to prevent interference. The microphone 131 mayalternatively work in conjunction with an attached audio device, such asbone conduction devices, to provide the user with audio feedback withoutbroadcasting the audio feedback.

The smart necklace 100 may improve social interactions. For example, thesmart necklace 100 may recognize faces in a room to identify potentialfriends, and provide the user with audio feedback identifying friends.The stereo cameras 121 and/or the camera 122 may be further able todetermine additional details about persons, such as moods orexpressions, or if they are engaging in physical activities, in order toalert the user. For example, the potential friend may extend a hand fora handshake or a “high five,” and the smart necklace 100 may use audioor haptic feedback to notify the user. The microphone 131 may recognizevoices of other persons to identify and appropriately notify the user,or may recognize a new voice to save for future identification.

Although the smart necklace 100 is described with respect to a blinduser, the smart necklace 100 may be used in other applications. Forexample, the smart necklace 100 may be used by peace officers and lawenforcement officers as a recorder which provides additionalenvironmental awareness. The smart necklace 100 may be further used byathletes to record sports in a real-time, first person view. Forexample, performing certain actions such as a swing can be recorded,including inertial motions, to analyze the motions. The smart necklace100 may also be used in hazardous environments to provide additionalsafety warnings. For example, the smart necklace 100 may be used in afactory to provide a factory worker additional warning about possiblehazardous conditions or obstacles. The smart necklace 100 can be amemory device to aid persons, such as Alzheimer's patients. The smartnecklace 100 can aid in shopping or otherwise navigating inventories byhelping to keep track of goods.

In such applications, the sensor 125 may be specifically chosen toprovide particularly relevant measurements. For instance, in anenvironment with harmful gas, the sensor 125 may detect dangerous levelsof gas and accordingly alert the user. The sensor 125 may providelow-light viewing, or the stereo cameras 121 and/or the camera 122 maybe capable of night vision, to provide the user with additionalenvironmental awareness and social interaction in low-light conditions,such as outdoors at night or photo-sensitive environments. The sensor125, the stereo cameras 121 and/or the camera 122 may be adapted todetect a light spectrum other than the visible light spectrum. Theantenna 142 may be an RFID or NFC reader capable of identifying RFID orNFC tags on goods.

In certain embodiments, the smart necklace 100 is designed toaccommodate blind or partially blind users. In such embodiments, alow-light viewing or night-vision camera (e.g., infrared camera) mayalso be utilized. For example, one camera 122 may be directed to normallighting and another camera 122 directed to night vision. For example, ablind user may be more likely to turn off the lights because he/she doesnot depend on the lighting. A night vision camera of the smart necklace100 may detect image data in the dark and provide the image data to theprocessor 111 for processing.

In some embodiments, the sensor 125 may be a light sensor for detectingan ambient light around the smart necklace 100. The processor 111 mayreceive the detected ambient light from the light sensor and adjust thestereo cameras 121 and/or the camera 122 based on the detected light,such as by adjusting the metering of the camera(s). This would allow thecamera(s) to detect image data in most lighting situations.

Because the smart necklace 100 may be used for environmental awarenessand social interaction, data detection of different light spectrums maybe useful. The visible light spectrum allows humans to detect certaindetails that other light spectrums may not provide. However, other lightspectrums may provide certain details that human visible light spectrumcannot provide. For example, details of certain objects may not beeasily detected by the visible light spectrum during a cloudy or foggyday. Another spectrum of light may provide better details of objectsduring these types of conditions. Spectrums in which the pair of stereocameras 121 and/or the camera 122 may be adapted to detect may includeextreme ultraviolet, near infrared, mid infrared, far infrared, etc. Formaximum efficiency and object detection, different sensors 125, stereocameras 121 and/or cameras 122 may be provided for detecting variouslight spectrum data. In some embodiments, a single camera 122 isprovided that detects a spectrum of light other than the visible lightspectrum, while the pair of stereo cameras 121 detects objects withinthe visible light spectrum.

FIG. 1B illustrates the processor 111 including multiple modules. Eachof the modules may perform a particular function, as will be describedherein. The processor 111 may include an object recognition module 150,a positioning/orientation detection module 152, an output determinationmodule 154, a map updating module 156, a mode selection module 158, afind module 160, an explore module 162, a scan module 164 and a capturemodule 166. The processor 111 may include all or some of these modulesand may include additional modules not illustrated in FIG. 1B. Forexample, the methods performed in the output determination module 154may be performed within the find module 160, the explore module 162, thescan module 164 and/or the capture module 166. The functionality of themodules will be described with reference to FIG. 12 through FIG. 22.

FIGS. 1A and 1B describe components of a smart necklace. One skilled inthe art will realize that the components described in FIG. 1A, as wellas any discussion thereof, may be applicable to other smart devices,such as a smart earpiece, smart glasses or the like. Similarly, anydiscussion of the modules of FIG. 1B are applicable to a smart necklaceor other smart devices, such as a smart earpiece, smart glasses or thelike.

FIG. 2 illustrates an embodiment of a smart necklace 200 viewed from thefront (i.e., along the Z axis). An XYZ axis is shown to illustrate theshape and relative position of components within the smart necklace 200.The smart necklace 200 includes an upper portion 201, a right portion208 and a left portion 209. The smart necklace 200 is to be worn arounda neck of a user. When worn on a user, upper portion 201 may rest on theback of a person's neck. The right portion 208 may extend over theuser's right shoulder such that a right end 215A of the smart necklace200 is positioned on or above the user's right chest. Similarly, theleft portion 209 may extend over the user's left shoulder such that aleft end 215B of the smart necklace 200 is positioned on or above theleft side of the user's chest.

The right portion 208 may include a right middle portion 202A, a lowerright portion 204A, and a button portion 206. In various embodiments,the right portion 208 may not be separated into the right middle portion202A, the lower right portion 204A and/or the button portion 206. Invarious embodiments, the lower right portion 204A and the button portion206 are combined into a single piece. The left portion 209 may include aleft middle portion 202B and a lower left portion 204B. In variousembodiments, the left portion may not be separated into the left middleportion 202B and the lower left portion 204B.

The upper portion may have a middle 270, a left end 272B and a right end272A. The upper portion 201 may be substantially straight at the middle270 and curved between the middle 270 and the ends 272 such that themiddle and lower portions may extend over the user's shoulder. The curvetowards the left end 272B and the right end 272A may be such that thecurves substantially mimic the user's neck and shoulders. This designallows the upper portion 201 to rest comfortably on the user's neck. Theupper portion 201 may be rigid, meaning that the upper portion 201 willnot bend or flex under normal pressure. This allows sensitive componentssuch as batteries, processors, memories or the like to be housed in theupper portion 201 without concern of the components becoming damaged.The upper portion 201 may be at least partially hollow such thatcomponents may be housed within the upper portion 201.

The upper portion 201 may include a power button 250. The power button250 may be positioned in the middle 270 of the upper portion 201. Whereused herein, if a component is positioned on a portion of the smartnecklace, then the component may be internal with reference to theportion, the component may be partially internal and partially externalwith reference to the portion or the component may be external to andcoupled to the portion. The power button 250 may be connected to aprocessor, such as processor 111, such that the power button may togglethe smart necklace 200 between an ON position and an OFF position. Whenin an ON position, components of the smart necklace 200 may receivepower from a power source, such as the battery 141.

In various embodiments, the processor may be adapted to determine astatus of the power supply. For example, the processor may be able todetermine a remaining operational time of the smart necklace 200 basedon the current battery status. In various embodiments, the processor maybe able to determine a percentage of power remaining in the battery. Thepower button 250 may be configured to send a power status request to theprocessor. For example, a user may depress and immediately release thepower button 250 to cause the processor 111 to determine a status of thepower source. In order to turn the smart necklace 200 to an OFF state,the user may depress and hold the power button 250 for a predeterminedamount of time. In various embodiments, a press and release may resultin an OFF state while a press and hold sends a request for power supplystatus. In various embodiments, a double click or any other clickpattern may be substituted for the press and release or press and hold.

The right middle portion 202A includes an upper end 203A coupled to theright end 272A of the upper portion 201 and a lower end 205A. The leftmiddle portion 202B includes an upper end 203B coupled to the left end272B of the upper portion 201 and a lower end 205B. The middle portions202 may be permanently coupled to the upper portion 201 or they may beremovably coupled to the upper portion 201. When the middle portions 202are removably coupled to the upper portion 201, the connection may besuch that the middle portions 202 will not detach from the upper portion201 under normal wearing conditions.

The middle portions 202 may be curved. This allows the middle portions202 to rest against the user's neck and/or shoulders. In someembodiments, the middle portions 202 may be constructed of a semi-rigidmaterial, such as rubber, silicone or the like. The semi-rigid materialmay bend or flex under certain forces but will return to its originalshape when the force has been removed. The semi rigid material may allowthe middle portions 202 to conform to the contours of an individualuser's shoulders. Thus, the semi rigid material of the middle portions202 allows the smart necklace 100 to fit comfortably to different users.

The right middle portion 202A may include a speaker 232A and a vibrationunit 233A. In various embodiments, the speaker 232A and the vibrationunit 233A may be the same device, such that a single device providesvibration data and audio data. In various embodiments, a cavity isformed in the right middle portion 202A such that a separate vibrationunit 233A and speaker 232A are positioned in the same cavity. In variousembodiments, the speaker 232A and the vibration unit 233A are positionedin separate locations on the smart necklace.

The left middle portion 202B similarly may include a speaker 232B and avibration unit 233B. The speaker 232B and the vibration unit 233B may bepositioned in a similar fashion as the speaker 232A and the vibrationunit 233A. By providing a speaker and/or a vibration unit on both sidesof the user, stereo information can be provided to the user. Forexample, a vibration by vibration unit 233A may indicate that the useris to turn right and a vibration by vibration unit 233B may indicatethat the user is to turn left. Alternatively, a vibration on vibrationunit 233A may instead indicate that the user is traveling too far to theuser's right and should therefore turn left and a vibration by vibrationunit 233B may indicate that the user should turn right.

The lower right portion 204A includes an upper end 207A coupled to thelower end 205A of the right middle portion 202A and a lower end 210A.The lower right portion 204A may be permanently coupled to the rightmiddle portion 202A or may be removably coupled to the right middleportion 202A. When the lower right portion 204A is removably coupled tothe right middle portion 202A, the connection may be such that the lowerright portion 204A will not detach from the right middle portion 202Aunder normal wearing conditions.

The lower right portion 204A may be substantially straight. Proximal tothe right end 215A, the lower right portion 204A may become larger inthe X direction as it approaches the right end 215A. This providesadditional surface area for components, such as buttons 252, to bepositioned towards the right end 215A of the smart necklace. The lowerright portion 204A may be constructed of a rigid material. The rigidmaterial may be at least partially hollow or contain a cavity such thatcomponents may be housed within the lower right portion 204A.

The lower right portion 204A may include a camera 222 and a microphone231. The camera 222 may be a single camera capable of detecting imagedata. The camera 222 may be adapted to detect image data of any lightspectrum including, but not limited to, the visible light spectrum, theinfrared spectrum, the near ultraviolet spectrum, etc. The camera 222may be a wide angle camera such that it can detect data at about 120degrees. The microphone 231 may be adapted to detect audio information.For example, the microphone 231 may detect speech of a user, speech ofanother person or speech of an environment of a user.

The lower left portion 204B includes an upper end 207B coupled to thelower end 205B of the left middle portion 202B and a lower end 210B thatis the same as the left end 215B of the smart necklace 200. The lowerleft portion 204B may be permanently coupled to the left middle portion202B or may be removably coupled to the left middle portion 202B. Whenthe lower left portion 204B is removably coupled to the left middleportion 202B, the connection may be such that the lower left portion204B will not detach from the left middle portion 202B under normalwearing conditions.

The lower left portion 204B may be similar to the lower right portion204A. The lower left portion 204B may become larger in the X directionas it approaches the left end 215B. This may provide a greater surfacearea for additional external-mounted components and/or a greater volumefor housing internal components. The lower left portion 204B may beconstructed of a rigid material and be at least partially hollow suchthat components may be housed within the lower left portion 204B.

The lower left portion 204B may include a pair of stereo cameras 221. Astereo camera 221A may be positioned proximal to the left middle portion202B while another stereo camera 221B is positioned proximal to the leftend 215B. The pair of stereo cameras 221 may be separated from eachother by a distance 217. The distance 217 may be selected based upon anoptimal range. For example, if it is determined that depth informationis most important between 5 and 10 feet, the distance 217 may be smallerthan if it is determined that an optimal distance for that depthinformation is between 10 and 15 feet. The stereo camera 221A and/or thestereo camera 221B may be wide angle cameras such that they can detectimage data at a range of about 120 degrees. The stereo cameras 221 maybe capable of detecting image data at various light spectrums,including, but not limited to, the visible light spectrum, the infraredspectrum, the near ultraviolet spectrum, etc.

The lower left portion 204B may also include a light sensor 225. Invarious embodiments, a single device may comprise the light sensor 225and the stereo camera 221B. In various embodiments, the lower leftportion 204B includes a cavity such that both the light sensor 225 andthe stereo camera 221B are positioned within the same cavity. In variousembodiments, the light sensor 225 and the stereo camera 221B may bepositioned at separate locations on the smart necklace 200. In variousembodiments, the light sensor 225 is coupled to each camera and stereocamera of the smart necklace 100. Coupling between the cameras and thelight sensor 225 may allow each camera to adjust its sensitivity tolight based on an ambient amount of light sensed by the light sensor225, such that each camera may detect an optimal quality of image data.In various embodiments, the processor may be coupled to the light sensor225 such that the processor may adjust image data received from thecameras based on the detected ambient light.

Between the lower right portion 204A and the right end 215A may be abutton portion 206. The button portion 206 has an upper end 213 coupledto the lower end 210A of the lower right portion 204A and a lower end214 that is the same as the right end 215A of the smart necklace 200.

The lower right portion 204A may be permanently coupled to the buttonportion 206 or may be removably coupled to the button portion 206. Whenthe lower right portion 204A is removably coupled to the button portion206, the connection may be such that the lower right portion 204A willnot detach from the button portion 206 under normal wearing conditions.In various embodiments, the lower right portion 204A and the buttonportion 206 are a single portion and/or the button portion 206 maysimply be an extension of the lower right portion 204A. In variousembodiments, the button portion 206 is removable such that a new orupdated button portion may be attached to the smart necklace 200. Inthis manner, functionality may be added to the smart necklace 200 byinclusion of the new button portion. The button portion 206 may be atleast partially hollow and constructed of a rigid material and thebutton portion may house components.

The button portion 206 may include a plurality of buttons. In theembodiment illustrated in FIG. 2, the button portion 206 includes abutton 252A, a button 252B, a button 252C and a button 252D. The buttons252 may be used as input to the smart necklace 200. The buttons 252 mayallow a user to select a mode of operation of the smart necklace 200.For example, each button may correspond to an operational mode of thesmart necklace 200, such that when a user depresses button 252A thesmart necklace 200 operates in a first mode, when a user depresses thebutton 252B, the smart necklace 200 operates in a second mode, etc.

Using the one-button-per-mode system provides simplified user input. Inmany situations, it may be preferred for the user to be able to switchmodes without drawing attention to herself. By learning the location ofthe buttons 252 and which mode of operation is associated with eachbutton, the user can quietly and easily select a preferred operatingmode without drawing attention of other people around him. Situationsmay also arise where it may be difficult for the smart necklace 200 tounderstand the user's voice over the ambient noise. Thisone-button-per-mode system prevents this issue, as no speaking may berequired of the user.

The smart necklace 200 may operate in at least four modes: explorermode, scan mode, find mode and capture. While in the explorer mode, thesmart necklace 200 provides data to the user associated with thesurroundings of the user. In some embodiments, the smart necklace 200may describe data collected by the stereo cameras 221, the camera 222and/or any other sensor to the user. In some embodiments, the smartnecklace 200 may only described data that is collected while the user ismoving (i.e., the field of view of the stereo cameras 221 and/or thecamera 222 is changing). The data may only be certain data, such ashazard data, whether a friend of the user is passing by, whether auser's favorite restaurant is detected, etc.

While in the scan mode, the smart necklace 200 may describe everythingthat is in the field of view of the stereo cameras 221, the camera 222and/or any other sensor. For example, the smart necklace 200 maydescribe everything in the field of view, such as by telling the userthat object X is at your 10:00, object Y is at your 11:00, objects Z andW are at your 12:00, etc. The smart necklace 200 may operate in the scanmode even if it is not in motion and/or being worn. For example, theuser could place the smart necklace 200 in a charging dock or in anyother position in which the smart necklace 200 could capture data withthe stereo cameras 221 and/or the camera 222. The smart necklace 200could then continue to describe information that is in the field of viewof the stereo cameras 221 and/or the camera 222.

While in the find mode, the smart necklace 200 can navigate the user toa desired object, place, person, etc. The user can provide data aboutthe desired object, place, person, etc., such as by speaking the name ofthe object, place, person, etc. The smart necklace 200 can thendetermine the location of the object, place, person, etc. and providenavigation directions to the user.

The capture mode may allow the smart necklace 200 to store its currentposition in the memory 112 so that it can guide the user back to thesame location at a later time. The capture mode may include 2instructions—capture and return. Capture stores the position information(and possibly any obstacles that may arise during a return trip to theposition) while return causes the smart necklace 200 to providenavigation instructions to the user for a return to the position. Invarious embodiments, a single press of the capture button may indicatethe capture instruction and a double click indicates the returninstruction.

This description of the modes of operation is not meant to be limiting.Explanation of the various modes and uses will be discussed in detailbelow with reference to FIGS. 18 to 26.

The smart necklace 100 illustrated in FIG. 2 is one embodiment of asmart necklace. One skilled in the art will realize that components of asmart necklace may be positioned other than illustrated in FIG. 2.

FIG. 3 illustrates the smart necklace 200 of FIG. 2 from the back (i.e.,along the Z axis). In FIG. 3, the smart necklace 200 is illustrated fromthe opposite side of the Z axis than illustrated in FIG. 2.

The upper portion 201 includes a back surface 269 substantially oppositea front surface (such as the front surface 368). The right middleportion 202A includes a back surface 244 substantially opposite a frontsurface (such as the front surface 385). The left middle portion 202Bincludes a back surface 245 substantially opposite a front surface (suchas the front surface 384). The lower right portion 204A includes a backsurface 246 substantially opposite a front surface (such as the frontsurface 388). The lower right portion 204A also includes an innersurface 266 and an outer surface 265 substantially opposite the innersurface 266. The lower left portion 204B includes a back surface 247substantially opposite a front surface (such as the front surface 370).The lower left portion 204B also includes an inner surface 264 and anouter surface 263 substantially opposite the inner surface 264. Thebutton portion 206 includes a back surface 248 substantially opposite afront surface (such as the front surface 389). The button portion 206also includes an inner surface 268 and an outer surface 267substantially opposite the inner surface.

The upper portion 201 of the smart necklace may include a battery 241.In various embodiments, the battery 241 may be centered within the upperportion 201 on the X axis. The battery 241 may be coupled to all of theelectronic devices within the smart necklace 200 such that the batterycan provide power to all electrical components within the smart necklace200.

The upper portion 201 may also include a processor 211. The processor211 may be coupled to all electronic components of the smart necklace200 such that the processor 211 can receive inputs and provide outputsfrom/to the electronic components. The upper portion 201 may alsoinclude a memory 212. The memory 212 may be coupled to the processorsuch that the processor 211 can store and retrieve data from the memory212. The memory 212 and the processor 211 may be positioned on the sameside or on opposite sides of the upper portion 201. It is preferred thatweight distribution of the upper portion 201 is centered in the middleof the upper portion 201 in the X direction. This will cause the weightof the upper portion 201 to be evenly distributed on the user,increasing the comfort of the smart necklace 200.

The lower left portion 204B may include an IMU 223, a binary switch 254,a toggle switch 256, an indent 260A and a connector 261A. The IMU 223may be similar to the IMU 123. The binary switch 254 may control a mutefunction of the smart necklace 200 such that when disengaged, thespeakers 232 can provide audio output and when engaged, the speakers 232may not provide audio output. The toggle switch 256 may correlate to avolume function of the smart necklace 200 such that when toggled in afirst direction, volume of audio from the speakers 232 becomes greaterand when toggled in a second direction, volume of output from thespeakers 232 becomes lower. The indent 260A may be an indent on the backside of the smart necklace 200. The indent 260A may include a connector261A. The connector 261A may be a snap connector, a magnetic connectoror other type of connector capable of physically and/or electricallyconnecting the connector 261A to another device.

The lower right portion 204A may include a GPS 224. The GPS 224 may besimilar to the GPS 124.

The button portion 206 may include an I/O port 243, an indent 260Bsimilar to the indent 260A, a connector 261B similar to the connector261A and a charging contact 262. In various embodiments, the lower leftportion 204B may include a charging contact within the indent 260A. TheI/O port 243 may be a 9 mm audio port, a USB port, a mini USB port orthe like. The charging contact 262 may be coupled to the battery 241such that the charging contact 262 may receive power and transfer thatpower to the battery 241 for storage. The charging contact 262 may beadapted to receive power via magnetic charging, inductive charging,direct charging or the like. In various embodiments, the chargingcontact 262 may be coupled to the processor 211 such that electronicdata may be transferred via the charging contact 262 in addition to orinstead of power.

The middle portions 202 may or may not include additional componentsthan those illustrated in FIG. 2. If no additional components arepresent in the middle portions 202, a connection 270A and a connection270B may exist within the middle portions 202 in order to electricallycouple the lower portions 204 to the upper portion 201. The connections270 may include a data bus, a power line, or any other electricalconnection. In some embodiments, the connections 270 may be replacedwith wireless connectivity between the lower portions 204 and the upperportion 201.

Because the upper portion 201 and the lower portions 204 may be hollowand rigid, electronic components may be safely positioned within theseportions. It may be desirable for bulky components such as the battery241 to be positioned in the upper portion 201. Because the upper portion201 is positioned adjacent a user's neck, additional weight may be moreeasily and comfortably supported in the upper portion 201 than the lowerportions 204. It is desirable that weight be evenly distributed betweenthe left middle portion 202B and the lower left portion 204B and theright middle portion 202A, the lower right portion 204A and the buttonportion 206. An even distribution of weight improves the comfort of thesmart necklace 200.

The left middle portion 202B and the right middle portion 202A may housecertain components. For example, the smart necklace 200 includesvibration units 233 and speakers 232 in the middle portions 202.Similarly, the smart necklace 200 may have an antenna 242 extend intothe left middle portion 202B. The antenna 242 may be coupled to theprocessor 211 such that the processor 111 may transmit and receivewireless signals via the antenna 142.

The antenna 242 may be wirelessly coupled to a device or devices remotefrom the smart necklace 200, such as a cloud 290, a mobile device 292, alaptop, a tablet or the like. In various embodiments, the cloud 290 mayinclude storage and/or processing that the smart necklace 200 mayutilize. For example, the smart necklace 200 may transmit certain datato the cloud 290 such that the cloud stores the data or processes thedata. The smart necklace 200 may later retrieve the stored and/orprocessed data from the cloud 290. In various embodiments, the smartnecklace 200 is designed to perform some functions locally, such as bythe processor 211, and is designed such that other functions areperformed remotely, such as by the cloud 290.

The mobile device 292 may be coupled to the smart necklace 200 such thatthe mobile device may perform some processing and storage functions forthe smart necklace 200. The mobile device 292 may also be connected tothe cloud 290 such that the mobile device 292 may perform some storageand/or processing functions and transmit additional storage and/orprocessing functions to the cloud 290. In various embodiments,processing and/or storage may be performed by any combination of thesmart necklace 200, the mobile device 292 and the cloud 290.

FIG. 4 illustrates a portion of the smart necklace 200 including some ofthe upper portion 201, the right middle portion 202A, the lower rightportion 204A and the button portion 206. The portion of the smartnecklace 200 is illustrated along the X direction. As illustrated, thesmart necklace 200 curves substantially 90 degrees about the Y axisbetween the upper portion 201 and the right end 315A. The button portion206 includes the I/O port 243 positioned on a surface that issubstantially perpendicular to the surface to which the buttons 252 areattached.

FIG. 5 illustrates a portion of the smart necklace 200 including some ofthe upper portion 201, the left middle portion 202B and the lower leftportion 204B. The portion of the smart necklace 200 is illustrated alongthe X direction. As illustrated, the smart necklace 200 is curved about90 degrees about the Y axis between the upper portion 201 and the leftend 315B. The curve about the Y axis allows the smart necklace 200 tobecome substantially flush with the back of the user's neck andsimultaneously with a user's shoulders and chest. As illustrated, thebinary switch 254 and the toggle switch 256 are positioned on a planesubstantially perpendicular to the plane on which the stereo camera 221Band the light sensor 225 are positioned. This allows for easy access ofthe binary switch 254 and/or the toggle switch 256 by the user.

FIG. 6 illustrates a smart necklace 300 according to variousembodiments. The smart necklace 300 may have the same or similardimensions and shape as the smart necklace 200. The smart necklace 300includes an upper portion 301, a right middle portion 302A, a lowerright portion 304A, a button portion 306, a left middle portion 302B anda lower left portion 304B. The smart necklace 300 is similar to thesmart necklace 200 except that a light sensor 325 is positioned near butseparate from a stereo camera 321B. In FIG. 6, the XYZ axis is againshown to illustrate the shape of the smart necklace 300 and the relativepositioning of components.

The upper portion 301 includes a bottom surface 386 and a top surface387 substantially opposite the bottom surface 386. A front surface 368may be substantially perpendicular to the bottom surface 386 and the topsurface 387. A back surface may exist substantially opposite the frontsurface. The bottom surface 386 and/or the top surface 387 may besubstantially smooth. The upper portion 301 is curved such that thebottom surface 386 may rest against a user's neck when the smartnecklace 300 is worn by the user. The bottom surface 386 and the topsurface 387 have a distance 361, which may be considered a width. Thedistance 361 may be substantially even across the upper portion 301 orit may vary depending on the location of the upper portion 301. Thebottom surface 386 and the top surface 387 may be separated by adistance 360, which may be considered a thickness. The distance 360 maybe substantially even across the upper portion 301 or it may varydepending on the location of the upper portion 301. The distance 361 maybe greater than the distance 360. This allows the smart necklace 300 tobe worn more comfortably by a user.

The right middle portion 302A includes a front surface 385. The frontsurface 385 has a distance 383, which may be considered a width. Thefront surface 385 may be a continuation of the top surface 387 such thatthey are the same surface. A back surface substantially parallel to thefront surface 385 is positioned a distance 382 (a thickness) from thefront surface 385. The distance 382 is smaller than the distance 383.This allows the back surface of the right middle portion 302A tocomfortably rest against a user's shoulders. Because the distance 382 isless than the distance 383, the smart necklace 300 may have a largersurface area in contact with a user's shoulders, increasing the comfortlevel of wearing the smart necklace 300.

The distance 382 may be similar to the distance 360, such that athickness of the smart necklace 300 is substantially similar throughoutthe smart necklace 300. The distance 383 may be substantially the sameas distance 361, such that the smart necklace 300 has a similar widththroughout. In various embodiments, the distance 383 may be smaller orlarger than the distance 361. The left middle portion 302B may besimilar to the right middle portion 302A, and have a front surface 384and a back surface substantially parallel to the front surface 384. Aswith the smart necklace 200, the middle portions may include speakers332 and vibration units 333.

The lower right portion 304A may have a front surface 388. A backsurface substantially parallel to the front surface 388 may exist adistance 305 (a thickness) from the front surface 388. The back surfaceis positioned against the user's body. The front surface 388 may have adistance 362 (a width). In various embodiments, the distance 362 may besubstantially the same throughout the lower right portion 304A or thedistance 362 may increase as the lower right portion 304A approaches thebutton portion 306. The distance 362 may be similar to the distance 383.The distance 305 may be similar to the distance 382. Accordingly, thedistance 305 may be less than the distance 362. Thus, the back surfacemay be larger than a side surface, increasing the comfort of the smartnecklace 300. As with the smart necklace 200, the lower right portion304A includes a camera 322 and a microphone 331.

The button portion 306 may include a front surface 389. A back surfacesubstantially parallel to the front surface 389 may exist a distance 392(a thickness) from the front surface 389. The button portion 306 mayinclude a distance 366 (a width). The distance 366 may be larger thanthe distance 362. This may allow more area on the front surface 389 forexternally-positioned components, such as buttons 352. The distance 366may be larger than the distance 392. This allows a larger surface areaof the button portion 306 to be positioned against a user, increasingcomfort of the smart necklace 300. As with smart necklace 200, thebutton portion 306 includes four buttons 352. In various embodiments,the button portion 306 may include more or less buttons.

In some embodiments, two buttons are raised from the button portion 306(in the positive Z direction) and two buttons are lowered from thebutton portion (in the negative Z direction). For example, the button352A and the button 352B may be positioned above (in the Z direction) aplane defined by a surface of the button portion 306 and the button 352Cand the button 352D may be positioned below (in the negative Zdirection) a plane defined by the surface of the button portion 306.This allows users to easily distinguish each button. Similarly, a notchis present in between each of the buttons 352 to increase the ability ofthe user to distinguish each button.

The lower left portion 304B may have similar dimensions to thecombination of the lower right portion 304A and the button portion 306,including a front surface 370 and a back surface substantially parallelto the front surface 370. In various embodiments, the dimensions of thelower left portion 304B may vary from the dimensions of the combinationof the lower right portion 304A and the button portion 306. As with thesmart necklace 200, the lower left portion 304B includes a stereo camera321A positioned proximal to the left middle portion 302B and a stereocamera 321B positioned distal to the left middle portion 302B. The lowerleft portion 304B also includes a light sensor 325 positioned distal tothe left middle portion 302B. The lower left portion 304B differs fromthe lower left portion 204B in that the light sensor 325 is positionedseparate from the stereo camera 321B.

In various embodiments, some or all components may be switched betweenthe lower right portion 304A and the button portion 306 and the lowerleft portion 304B. In various embodiments, the button portion 306 may bepositioned adjacent the lower left portion 304B instead of the lowerright portion 304A. The button portion 306 is such named because buttons352 are positioned on the button portion 306. In various embodiments,the button portion 306 may include additional components and/or may notinclude the buttons 352.

A distance 397 exists between the lower right portion 304A and the lowerleft portion 304B proximal to the middle portions 302. A distance 398exists between the right end 315A and the left end 315B. In variousembodiments, the distance 398 and the distance 397 may vary based onforces applied to the semi-rigid middle portions 302. Accordingly, thedistance 397 and the distance 398 may increase in order to allow a userto put the smart necklace 300 around the user's neck and then thedistances may decrease once the smart necklace 300 is positioned on theuser's neck. In a resting position (i.e., without external force appliedto the smart necklace 300), the distance 397 may be greater than thedistance 398. This may allow the right end 315A to begin to approach theleft end 315B, thus allowing the smart necklace 300 to become moresecure to a user.

In various embodiments, each of the portions of the smart necklace 300may have a width that is substantially larger than a thickness. Forexample, the width of each part may be between two and twenty times aslarge as the thickness of each part. In various embodiments, the widthof each part may be between 3 and 6 times as large as the thickness ofeach part. The ratio of width to thickness results in each part beingsubstantially flat, causing the smart necklace 300 to be comfortablewhen worn by the user.

FIG. 7 illustrates a view of the smart necklace 300 from the bottom(i.e., along the Y axis). As illustrated in FIG. 5, the distance 366 isgreater than the distance 392. In various embodiments, the distance 366may be two, three, four or more times as large as the distance 392. Thesame or a similar relationship may exist between distance 361 anddistance 360. In various embodiments, the distance 361 may be greaterthan the distance 366. This allows larger (and thus potentially heavier)components to be positioned in the upper portion 301 than the buttonportion 306. This may be ideal for optimal comfort and balancing ofcomponents within smart necklace 300.

The distance 366 may be greater than the distance 362. It may be easierfor a user to interact with the smart necklace 300 at the right end 315Aand/or the left end 315B than a position more proximal to the upperportion 301. Because of this, the larger distance 366 may provide moresurface area for user interaction proximal to the right end 315A and/orthe left end 315B.

FIG. 8A illustrates a portable charging unit 500 for use with a smartnecklace, such as smart necklace 100, 200 and/or 300. The portablecharging unit 500 may be relatively lightweight such that a user of asmart necklace may also easily carry the portable charging unit 500 in apurse or pocket. The portable charging unit 500 may be used to rechargea battery of the smart necklace or provide additional power to the smartnecklace. The portable charging unit 500 includes a battery 502, a strap506 coupled to the battery 502, a connector 512 and a charging contact510 coupled to the strap 506, a connection 508 coupled to the chargingcontact 510 and the battery 502, and an indent and connector 504.

The battery 502 may be adapted to receive, store and discharge energy.The charging contact 510 may be positioned on a charger such that energymay be received by the charging contact 510 and travel to the battery502 for storage via the connection 508. The connector 512 may be a snapconnector, a magnetic connector or the like, and may be adapted toattach the strap 506 to a charging device. The indent and connector 504may include a snap connector, a magnetic connector or the like such thatthe connector 512 may attach to the indent and connector 504 for ease ofstorage. The battery 502 may be any type of battery capable of storing acharge. The strap 506 may be a malleable material such that theconnector 512 and the charging contact 510 may be easily positionedadjacent a connector and charging contact of the smart necklace.

FIG. 8B illustrates how the portable charging unit 500 may connect toand charge a smart necklace 200. As illustrated, the button portion 206includes the indent 260B, the connector 261B and the charging contact262. The charging contact 262 of the smart necklace may contact thecharging contact 510 of the portable charging unit 500. Power may betransferred between the portable charging unit 500 and the smartnecklace 200 via the charging contact 262 and the charging contact 510.The connector 512 of the portable charging unit 500 is adapted to attachto the connector 261B of the smart necklace 200 such that the chargingcontact 262 remains adjacent to or in contact with the charging contact510. The connector 512 is adapted to be positioned within the indent 260to accommodate a better physical connection between the smart necklace200 and the portable charging unit 500. This increases the likelihoodthat the portable charging unit 500 will not separate from the smartnecklace 200 until so desired.

In various embodiments, the charging contact 262 may be electricallycoupled to the processor 211 and be adapted to receive and transmit datasignals. This allows the smart necklace 200 to connect to a mobiledevice, a computer, a tablet or the like via the charging contact 262.The smart necklace 200 may be adapted to transmit data to and receivedata from the mobile device, laptop, tablet, portable charging unit 500or the like via the charging contact 262.

In various embodiments, it may be desirable for the ends of a smartnecklace to be attached together to provide a more secure fit of thesmart necklace to a user. In various embodiments, this connection mayinclude an electrical connection such that data may transfer between thetwo ends without traveling through the upper portions.

FIG. 9A illustrates a strap attachment 600 that is adapted to attach thedistal ends of a smart necklace to each other. The strap attachment 600includes a strap 604, a connector 602A and a connector 602B. In variousembodiments, the connectors 602 may include communication contacts 606.In these embodiments, a connection 608 may exist between thecommunication contacts 606. The connectors 602 may be snap connectors,magnetic connectors or the like. The strap 604 may be made of anyflexible, semi-rigid or rigid material. The communication contacts 606may be a contact of capable of transmitting and receiving data signals.The connection 608 may be a cable, a wire, a wireless connection or thelike that is capable of facilitating data transfer.

FIG. 9B illustrates strap attachment 600 connected to the smart necklace200. In the embodiment illustrated in FIG. 9B, a charging contact 262Ais positioned within the indent 260A. As illustrated, the connector 602Amay attach to the connector 261A and the connector 602B may attach tothe connector 261B. In this manner, the strap attachment 600 may provideadditional support to increase the likelihood that the smart necklace200 will remain in the desired position on a user.

In various embodiments, the communications contact 606B may become incontact with the charging contact 262B, and the communication contact606A may be in contact with the charging contact 262B. In this manner,the charging contacts 262 may transmit and receive data via thecommunication contacts 606. This data may transfer between the chargingcontacts 262 via the connection 608 of the strap attachment. This mayallow components at the ends of the smart necklace 200 to transfer dataover a shorter connection distance, allowing faster communicationsbetween the components.

It may be desirable to change the size of a smart necklace in order tobetter fit people having different dimensions. FIG. 10A illustrates afirst insert 870 that is configured to be attached to a smart necklacein order to enlarge the size of the smart necklace. The first insert 870includes a body 872 that may have the same dimensions in the X directionand the Z direction as a middle portion and/or a lower portion. Thefirst insert 870 may have a relatively small distance in the Y directionsuch that it does not drastically alter the dimensions of the smartnecklace when attached.

The first insert 870 also includes a proximal connector 874 and a distalconnector 876. A connection 878 may exist between the proximal connector874 and the distal connector 876 such that data may transfer between theproximal connector 874 and the distal connector 876 via the connection878. In various embodiments, the proximal connector 874 and the distalconnector 876 may include a mechanical connector and an electricalconnector, such that the proximal connector 874 and the distal connector876 may allow the transfer of data as well as secure the first insert870 to the smart necklace. In various embodiments, separate electricaland mechanical connectors may exist instead of the proximal connector874 and the distal connector 876.

FIG. 10B illustrates a second insert 880. The second insert 880 issimilar to the first insert 870 except the second insert 880 has alarger distance in the Y direction. The second insert 880 includes aproximal connector 884, a distal connector 886, a body 882 and aconnection 888. The second insert 880 may have a larger distance in theY direction than the first insert 870 so that a smart necklace may bemore accurately constructed to fit a user. Otherwise, the second insert880 is similar to the first insert 870.

FIG. 10C illustrates a smart necklace 800 including a first insert 870and where the upper portion 801 is disconnected from the left middleportion 802B. As illustrated, the first insert 870 is positioned betweenthe upper portion 801 and the right middle portion 802A. This allows adistance from the upper portion 801 to the right end 215A to be larger,enabling the smart necklace 800 to be worn by a person having largerdimensions. In various embodiments, the first insert 870 may attach tothe upper portion 801 and/or the right middle portion 802A via screws,snap connectors or the like. It is preferred that the proximal connector874 and the distal connector 876 provide a sufficiently strongconnection such that the right middle portion 802A and the upper portion801 do not disconnect from the first insert 870 during normal use of thesmart necklace 800.

The upper portion 801 and the left middle portion 802B are disconnectedto illustrate how an insert, such as the first insert 870, can attach tothe smart necklace 800. The upper portion includes a proximal connector860 at one or both ends of the upper portion 801. The left middleportion 802B includes a distal connector on the upper end of the leftmiddle portion 802B. The proximal connector 860 and/or the distalconnector 862 may be mechanical and/or electrical connectors. In variousembodiments, separate mechanical and/or electrical connectors may existinstead of the proximal connector 860 and the distal connector 862.

The proximal connector 874 of the first insert 870 is adapted to attachto the proximal connector 860. The distal connector 876 of the firstinsert 870 is adapted to attach to the distal connector 862. Data maytransfer between the proximal connector 860 and the distal connector 862via the proximal connector 874, the distal connector 876 and theconnection 878. In this manner, the first insert 870 provides amechanical and electrical connection between the upper portion 801 andthe left middle portion 802B.

Similarly, when no insert is included with the smart necklace 800, theleft middle portion 802B may attach to the upper portion 801 via theproximal connector 860 and the distal connector 862.

The smart necklace 800 differs from the smart necklace 200 and the smartnecklace 200 in that separate speakers 832 and vibration units 833 areprovided. In the smart necklace 800, the vibration unit 833A ispositioned substantially adjacent the speaker 832A. In variousembodiments, the vibration unit 833A may be positioned closer or fartherto the speaker 832A than illustrated in FIG. 10C. The smart necklace 800also differs from the smart necklace 200 and the smart necklace 200 inthat no button portion is present on the smart necklace 800. The smartnecklace 800 may include the same dimensions as the smart necklace 200and the buttons may be positioned on the lower right portion 804A.

FIG. 11A illustrates a charging unit 400 configured to charge a smartnecklace, such as smart necklace 200. The charging unit 400 includes abase 402, a post 410A and a post 410B. The posts 410 extend away fromthe base and each include a connector 404. The connectors 404 may besnap connectors, magnetic connectors or the like. The post 410B includesa charging contact 406 that is adapted to charge the smart necklace. Thecharging unit 400 also includes a power line 408 that is adapted totransmit power from a source, such as a wall outlet, to the chargingcontact 406. The base 402 is adapted to rest on a substantially flatsurface such that the posts 410 extend upward from the substantiallyflat surface. The base provides sufficient support that when a smartnecklace is charging on the charging unit 400, the charging unit 400remains in the upright position.

FIG. 11B illustrates the smart necklace 200 positioned on the chargingunit 400. The connector 404A of the charging unit 400 is adapted toattach to the connector 261B, and the connector 404B of the chargingunit 400 is adapted to attach to the connector 261A. This allows thesmart necklace 200 remain attached to the charging unit 400 whilecharging. In preferred embodiments, the base 402 is adapted to hold thesmart necklace 200 in an upright position, as illustrated in FIG. 11B.The charging contact 406 is adapted to allow power to transfer from thepower line 408 to the smart necklace 200 via the charging contact 262.In various embodiments, the charging unit 400 may attach to anotherdevice, such as a mobile device, a computer, or the like, via the powerline 408 or another electrical connection. In this manner, data maytransfer between the portable device, computer, or the like and thesmart necklace 200 via the charging contact 406 of the charging unit 400and the charging contact 262.

In various embodiments, the charging unit 400 is adapted to cause thesmart necklace 200 to be positioned in an upright position such as thatillustrated in FIG. 11B. A user may leave the smart necklace in an ONposition while the smart necklace is connected to the charging unit 400.In this manner, the smart necklace 200 may provide environmentalawareness and social interaction to a user while positioned on thecharging unit 400.

FIG. 12 illustrates a method 1200 for updating location information in amemory, such as memory 112. The method 1200 may be performed by aprocessor, such as processor 111. The method 1200 may be performedwithin the map updating module 156. The method 1200 may be performed ona smart device, such as the smart necklace 100, or another smart device,such as a smart clip, smart glasses or the like.

In block 1202, the smart necklace 100 may arrive at a new location. Thelocation may be an outdoor region, an indoor facility such as a mall orshopping center, or the like. In block 1204, it is determined whether amap of the new location is available. The processor 111 may search thememory 112 to determine if a map is available within the memory 112. Ifa map is not available in the memory 112, then the processor 111 may,via the antenna 142, search a remotely-connected device and/or the cloudfor a map of the new location. The map may include any type of locationinformation, such as image data corresponding to a location, GPScoordinates or the like.

In block 1205, if a map is available, then the processor 111 mayretrieve the map. In various embodiments, the processor 111 may storethe map in the memory 112 if the map was retrieved from a remote deviceor the cloud. If no map is available, the processor may inform the uservia the interface array 130 that no map is available in block 1206. Inthis way, the user can know that no map data is available. This allowsthe user to search for a map of the area if he so desires.

In block 1208, data such as location information, such as from the IMU123 and/or the GPS 124, and/or image data, such as from the pair ofstereo cameras 121 and/or the camera 122, may be detected. This data maybe used to identify the current location and the surroundingenvironment. For example, image data can be parsed into detectableshapes such as doors, store signs or the like, and the data may beassociated with a position on a map. If a map of the location had beenfound, this newly detected data may be compared to data associated withthe retrieved map. The processor 111 may compare the detected image datato the image data associated with the map. If the detected image datamatches the stored image data, then the processor 111 may determine theposition of the smart necklace 100 based on the match. If the smartnecklace is moved to a new location, the processor 111 may receivepositioning data from either the IMU 123 or the GPS 124. The processor111 may be able to determine the new location of the smart necklace 100based on the previous location and the positioning data.

If the data associated with the position on the retrieved map isincorrect, the processor 111 may replace the incorrect data with thenewly detected data. The smart necklace 100 and other smart devices mayshare a map. For example, the map may be accessible via the cloud. Aseach smart device detects objects in an area covered by a shared map,each smart device may update the map, such that the aggregation ofupdates by the smart devices results in an accurate map.

If no map is available, the processor 111 may create a map within thememory 112, the cloud and/or the remote device. The new map may becontinuously updated as new data is detected, such that a map of thelocation including associated data can be generated based on thedetected data.

In some embodiments, a physical map of the new location may be present.For example, some malls include directories having map information. Thesmart necklace 100 may detect this visual map and create a new mapwithin the memory 112 or the cloud based off of the physical map. Invarious embodiments, the physical map may also include instructions fordownloading a virtual map of the area. For example, the instructions mayinclude a web address, a quick response code (QR code), or the like. Asmart necklace 100 may access the web address, scan the QR code, etc. tocause the map to be downloaded to the memory 112 or the cloud such thatthe smart necklace 100 can access the new map.

FIG. 13 illustrates a method 1300 for selection of a mode of a smartdevice, such as smart necklace 100. The method 1300 may be performedwithin the mode selection module 158. In block 1302, a mode selection isreceived. This mode selection may be received via the input device 134.For example, the smart necklace 100 may include a plurality of buttonssuch that each button is associated with a mode. The user may depress aparticular button to cause the associated mode to operate. In variousembodiments, the mode selection is performed based on input from themicrophone 131. For example, the user may say a command that isassociated with a particular mode or sub-mode. In response to receivingthe command, the smart necklace 100 may operate in the associated modeor sub-mode. In various embodiments, the smart necklace 100 may includea touch screen such that mode selection may be performed using the touchscreen. In various embodiments, mode selection may be determined basedon visual data from the stereo camera 121 and/or the camera 122. Forexample, a particular gesture by the user may indicate selection of amode or sub-mode, such that upon detection of the gesture, the smartnecklace will operate in the associate mode.

In block 1304, if the selected mode is the find mode, the method mayproceed to Location A. In block 1306, if the selected mode is exploremode, the process may proceed to Location B. In block 1308, if theselected mode is the scan mode, then the process may proceed to LocationC. In block 1310, if the selected mode is the capture mode, then theprocess may proceed to Location D.

FIG. 14 illustrates a method 1400 to be performed if the selected modeis the find mode. The method 1400 may be performed in the find module160. In block 1402, a request is received that includes a desiredobject, place, or person. This request may be a verbal command, such as“navigate to Macy's,” “where is Belks,” “take me to the exit,” or thelike. In various embodiments, the request may be input by the user viathe input device 134, such as by typing on a keypad or a touchscreen.

In block 1404, the processor 111 determines whether the desired objecthas been located in the memory 112. In order to locate the desiredobject, person, or place, the processor 111 may search data in anaccessible memory for the object, person or location. For example, thedata may include a store directory within a mall. If the desired objectis a store, such as Sears, the desired object is found if a mapincluding data associated with Sears is found. Additionally, theprocessor may access the memory 112 and/or the cloud to determine anyinformation associated with the desired object, person or place, such asimage data, location data, or the like. For example, the data mayinclude an image of Sears. If the desired object has been located inmemory, data associated with the desired object, person or place will beretrieved by the processor 111. The data may include any map data, imagedata, location data or the like.

In block 1406, if the desired object is not located, then the processor111 may inform the user via the interface array 130. For example, aparticular combination of tones and/or vibrations may be provided to theuser. Also, the speakers 132 may output audio, such as “[the desiredobject] is not found.” In various embodiments, the smart necklace 100may request additional information from the user. For example, the smartnecklace 100 may request another name for the desired object, person orplace. In various embodiments, the processor 111 may search a memory inthe cloud or may search the internet for map data or other dataindicating the location of the desired object, person or place.

In block 1408, if the desired object, person or place has been located,then the processor 111 determines whether navigation has been requested.In some embodiments, this may be determined based on the request fromblock 1402. For example, if the user says “navigate to Macy's,” it maybe determined in block 1408 that navigation is requested. However, ifthe user says “where is Belks,” the smart necklace 100 may determinethat navigation is not requested. The smart necklace 100 may be designedsuch that certain words indicate navigation requests and certain wordsindicate no navigation request.

The smart necklace 100 may also learn preference data of the user. Forexample, during previous iterations of method 1400 within a particulararea, the user may have never requested navigation. After a certainnumber of iterations within the particular area without navigationrequests, the processor 111 may determine that the user does not desirenavigation within this particular area.

In block 1410, if navigation was not requested, then the smart necklace100 may determine the location of the desired object, person or place ona map. The smart necklace 100 may also determine the location of thesmart necklace on the map such that directional instructions between thesmart necklace 100 and the object, person or place may be determined. Invarious embodiments, the smart necklace 100 may determine the locationof the object, person or place relative to the smart necklace 100instead of or in addition to utilizing the map.

In block 1412, the location information may be provided to the user viathe interface array 130. The location information may be consideredangular information, as the location information includes a direction(angular) in which the desired object, person or place is located. Thedirectional information may be provided in a clock-face type,directional type or the like. This information may be provided via thespeaker 132 and/or the vibration unit 133. The location information maybe provided in multiple manners, and the particular manner may bedetermined based on user preferences. For example, the smart necklace100 may provide output such as “Macy's is 200 yards to your 2 o'clock,”“Belks is upstairs to your left,” “Sears is 100 yards forward and 200yards to the left,” a particular vibration pattern or the like.

Returning to block 1414, if it is determined that navigation isrequested, the location of the desired object, person or place isdetermined and the location of the smart necklace 100 is determined. Theprocessor 111 may search data in the memory 112, such as map data, todetermine the location of the desired object, person or place. In someembodiments, the processor 111 may recognize objects within the FOV ofthe pair of stereo cameras 121 and/or the camera 122. The processor 111may determine if the desired object, person or place is one of therecognized objects.

The location of the smart necklace may be determined using a map andpositioning data, such as from the IMU 123 and/or the GPS 124. Imagedata from the pair of stereo cameras 121 and/or the camera 122 may beutilized instead of or in addition to the positioning data. In someembodiments, a map may include image data such that the processor 111may compare detected image data to the image data associated with themap and determine the location of the smart necklace 100 based on theimage data. In various embodiments, the processor 111 may determinepositioning based on any combination of positioning data from the GPS124, positioning data from the IMU 123, image data from the pair ofstereo cameras 121, image data from the camera 122 and sensed data fromthe sensor 125. Inertial movement data, such as from the IMU 123, may beused by the processor 111 to track the current location of the smartnecklace as the user moves.

In addition, data collected using the GPS 124 can enhance identificationof data collected by the camera 122. For example, if the camera 122provides an image of the building, the processor 111 can determine ifthe building is detected correctly by utilizing data regarding thelocation of the user in the world, because building types differ indifferent parts of the world.

The GPS information may be inadequate because it may not providesufficiently detailed information about the surrounding environment.However, the GPS information can be utilized along with visual data fromthe camera 122 to draw inferences that are helpful to the user. Forexample, if the GPS information indicates that the smart necklace 100 iscurrently inside a building, and the camera 122 provides informationregarding an object, the processor 111 can limit its search to objectsthat would rationally be inside the building. For example, if an imageprovided by the camera 122 appears like a truck, the processor 111 canrule out the possibility that the object is a truck based on the GPSinformation. In other words, it is more likely to be an image of aposter of a truck, because the poster can rationally be within abuilding and a truck cannot. The GPS 124 provides location information,which along with the inertial guidance information, including velocityand orientation information provided by the IMU 123, allows theprocessor 111 to help direct the user.

In block 1416, the smart necklace 100 may determine navigationinstructions between the smart necklace 100 and the desired object,person or place. The navigation instructions may be referred to asangular information because they include at least one direction (anangle) in which the user should move to reach the desired object, personor place. The angular information may be given in a clock-face format,an angular format (e.g., 20 degrees to the right) and/or directionalformat. The navigation instructions may be determined based on map dataand/or data detected by the sensor array 120. In block 1416,non-traversable regions may be detected. These regions may be stored inthe map data and/or may be detected by the sensor array 120. An exampleof a non-traversable region would be an area in which a wall exists.

The navigation instructions include a path over which the user maytravel, and exclude the non-traversable regions if the processor hasdetected these regions. The navigation instructions may be furthermodified for the user's needs. For example, a blind person may preferroutes that follow walls. Using the IMU 123 and/or the GPS 124 and othersensors, the smart necklace 100 can determine the user's location andorientation to guide them along the path, avoiding obstacles.

In block 1418, navigation instructions may be provided to the user viathe interface array 130. While travelling along the path, the smartnecklace 100 may inform the user about signs or hazards along the path.The vibration unit 133 and/or the speaker 132 provide audio and hapticcues to help guide the user along the path. For example, the speaker 132may play a command to move forward a specified distance. Then, specialaudio tones or audio patterns can play when the user is at a waypoint,and guide the user to make a turn by providing additional tones or audiopatterns. A first tone, audio pattern or vibration can alert the user tothe start of a turn. For example, a single tone or a vibration from aleft smart necklace may indicate a left turn. A second tone, audiopattern or vibration can alert the user that the turn is complete. Forexample, two tones may be provided, or the vibration may stop so that aleft device ceases to vibrate, when the turn is complete.

The navigation instructions may be provided in various manners. Forexample, all instructions may be initially provided to the user and thenupdated or corrected as the user proceeds along the route. In someembodiments, instructions are provided as the user traverses the route.In some embodiments, navigation instructions may be provided using clockface directions, such as “turn to your 2 o'clock, proceed 20 yards, turnto your 11 o'clock, proceed 10 yards,” or by providing directioninformation, such as “turn to your right, proceed 20 yards, turn to yourleft, proceed 5 yards,” or the like. In various embodiments, navigationinstructions may be provided via tones and/or vibrations. For example,the smart necklace 100 may vibrate and/or play a tone on the right toindicate a right turn. In various embodiments, the smart necklace 100may vibrate and/or play a tone on the right to indicate a left turn. Inthese embodiments, the vibrations and/or tones may act like bumpers tokeep the user on a desired path.

Different tones or patterns may also signify different degrees of turns,such as a specific tone for a 45 degree turn and a specific tone for a90 degree turn. Alternatively or in addition to tones and vibrations,the smart necklace 100 may provide verbal cues, similar to a car GPSnavigation command. High level alerts may also be provided through audiofeedback. For example, as the smart necklace 100 reaches a predetermineddistance—such as a foot or other value which may be stored in the memory112 and may be adjusted—from an obstacle or hazard, the speaker 132and/or the vibration unit 133 may provide audible alerts. As the smartnecklace 100 gets closer to the obstacle, the audible alerts and/orvibrations may increase in intensity or frequency.

In block 1420, it is determined whether the smart necklace 100 is at thedesired object, person or place. This may be determined by comparingdetected location data, positioning data and image data to data storedin the memory 112. If the detected data matches the data stored in thememory, the processor 111 may determine that the smart necklace 100 isat the desired object, person or place.

In block 1422, if the processor 111 has determined that the smartnecklace 100 is at the desired object, person or place, then the smartnecklace 100 may alert the user via the interface array 130. Forexample, the smart necklace 100 may output audio that says “arrived atMacy's.” In various embodiments, the smart necklace 100 may output aspecial vibration pattern and/or tone to indicate that the user hasarrived at the desired object, person or place.

In block 1424, it is determined whether the user is still on thenavigation route. If the user is still on the navigation route, then theprocess may return to block 1418. If the user is not still on thenavigation path, then the process may return to block 1416 where newnavigation instructions between the smart necklace 100 and the desiredobject, person or place may be determined.

While navigating the user to the desired object, person or place, theprocessor 111 may determine obstacles in the path of the user. Theprocessor 111 may use positioning data to determine the path of the userand image data and/or positioning data and map data to determine theobstacle. The processor 111 may determine that an obstacle is in thepath of the user by identifying an object based on image data anddetermining that the current path of the user will cause the user tocollide with the obstacle. The processor 111 may also compare thecurrent position and path of the user to map data. The map data mayindicate an obstacle along the current path.

In response, the smart necklace 100 may alert the user to theseobstacles in the user's path and/or navigate the user around theobstacles. For example, the smart necklace 100 may provide informationto the user if an obstacle is directly in front of the user within apredetermined distance, such as 6 feet. The smart necklace 100 may, inresponse to determining an obstacle within the predetermined distance ofthe user, provide output via the speaker 132 such as “obstacle directlyahead 4 feet.” In various embodiments, the navigation instructions maybe altered so that the navigation instructions direct the user aroundthe obstacle. In various embodiments, a particular tone and/or vibrationpattern may be provided to the user indicating the obstacle.

FIG. 15 illustrates an exemplary implementation of the method 1400. InFIG. 15, a user using a smart device, such as the smart necklace 100, isin a room 1501. While the user is at a location 1500, the user mayselect the find mode and request navigation instructions to the women'srestroom. The smart necklace 100 may load a map of the room 1501 intothe memory 112 such that it is accessible by the processor 111. The mapmay include various levels of detail. For example, the map may includethe location of restrooms, sub-rooms 1504, desks 1508 and 1511, chairs1510 and the like. In various embodiments, the map may only includefixed structures such as the sub-rooms 1504 and the restrooms.

After accessing the map, the smart necklace 100 may determine thelocation of the women's restroom 1502 and the location of the smartnecklace 100. The processor 111 may determine navigation instructionsbased on the location of the women's restroom 1502 and the location ofthe smart necklace 100. The smart necklace 100 may then proceed toprovide navigation instructions to the user. In some embodiments, thesmart necklace 100 may determine a route to the women's restroom 1502that avoids known obstacles. The smart necklace 100 may alter thesenavigation instructions based on detected objects along the route. Insome embodiments, the smart necklace 100 may instruct the user to simplywalk towards the women's restroom 1502 and alert the user to obstaclesalong the way.

In FIG. 15, the smart necklace 100 is aware of the sub-rooms 1504 andthe restrooms 1502 and 1503. As the user begins walking along path 1512,the smart necklace 100 may detect the table 1508A directly ahead of theuser. The smart necklace 100 may instruct the user to turn to the rightso that the user walks along path 1514. When the user reaches position1515, the smart necklace 100 may detect the table 1508B. In response,the smart necklace 100 may instruct the user to turn left and walk alongpath 1516. When the user reaches position 1517, the smart necklace 100may detect the chairs 1510C, 1510D, 1510A, 1510B, 1510E and 1510F. Inresponse, the smart necklace 100 may instruct the user to turn right andwalk along path 1518. When the user reaches position 1519, the smartnecklace may detect the chairs 1510G, 1510H, 15101, and 1510J andinstruct the user to turn left and follow path 1520.

When the user reaches position 1521, the smart necklace 100 maydetermine that the user should turn right in order to reach the women'srestroom 1502. In response, the smart necklace 100 may instruct the userto turn right and follow path 1522. When the smart necklace 100 reachesposition 1523, it may instruct the user to turn right and follow path1524.

When the user reaches position 1525, the smart necklace 100 may instructthe user to turn left and follow path 1526. When the user reachesposition 1527, the smart necklace 100 may determine that the door to thewomen's restroom 1502 is positioned directly to the left of the user. Inresponse, the smart necklace 100 may instruct the user to follow path1528 and inform the user that the door to the women's restroom 1502 isimmediately ahead of the user.

FIG. 16 illustrates a method 1600 to be performed when a smart device,such as the smart necklace 100, is in the explore mode. The method 1600may be performed by the explore module 162.

In block 1602, the processor 111 may determine settings of the smartnecklace 100. These settings may be stored in the memory 112 and will bediscussed with reference to FIG. 22. The data within the settingsdiscussed with reference to FIG. 22 may be applicable to any methodsperformed by the device. The settings determined by the processor 111may include a granularity (i.e., scope of information) setting. Thescope of information setting may determine an amount of granularity thatthe smart necklace 100 will provide to the user. For example, the usermay desire to only receive information associated with large and/orimportant objects, may desire to receive information associated with anyand all objects, or anywhere in between the two.

In block 1604, it is determined whether constant information isrequested by the user. In various embodiments, the user may verbally saywhether constant information is desired. This audio information may bedetected by the microphone 131 such that the processor 111 can determinethe desires of the user. In various embodiments, the input device 134,such as a button, may be depressed and released instantly to indicatethat constant information is or is not requested and may be depressedand held to indicate the other. In various embodiments, a single clickmay indicate one constant information or not and a double click mayindicate otherwise.

In block 1606, if it is determined that constant information is notrequested, the smart necklace 100 may detect objects and/or peoplewithin a pre-determined distance and angle of the smart device. Theobjects and/or people may be detected by the camera 122 and/or the pairof stereo cameras 121. For example, the pre-determined distance may be15 feet and the pre-determined angle may be 20 degrees, such that datais detected within a 20 degree view and 15 feet distance of the smartnecklace 100. The predetermined distance and angle may be preloaded inthe smart necklace 100 or may be a preference that the user can set. Bylimiting the detected objects to this predetermined distance and angle,objects may be detected within an optimal viewing range. Some distances,such as beyond 30 feet, may not be of significant importance to a blinduser. Similarly, objects very far on either side of the user may not beof significant importance. By limiting the viewing range, the user mayreceive the more important information.

After detecting the objects within the pre-determined distance andangle, the processor 111 may compare the detected objects to image datain the memory 112 in order to identify the detected objects. The smartnecklace 100 may compare the detected objects to image data from varioussources. In various embodiments, the smart necklace 100 may compare adetected person to images of Facebook friends of the user to identifythe detected person. In various embodiments, the smart necklace 100 maycompare a detected object to objects on the internet to identify theobject.

After detecting and identifying the objects and/or people within thepre-determined distance and angle, information identifying the objectsand/or people is provided to the user via the interface array 130. Theinformation provided to the user may include the name of the objectand/or person, the direction of the object and/or person and/or thedistance to the object and/or person. The information may vary based onthe scope of information setting. For example, if the scope ofinformation setting is highly detailed, then a lot of detailedinformation may be provided to the user. This detailed information mayinclude the name of the objects and/or people and descriptions of theobjects and/or people. For example, the smart necklace 100 may provideoutput saying “blond female 8 feet ahead at your 11 o'clock, 3 foot hightable 7 feet ahead at your 12 o'clock and 3 foot diameter trash can atyour 1 o'clock.” If the scope of information setting is low, then thesmart necklace 100 may only provide the name of the object and/or peopleand may or may not include direction and distance information. Forexample, the smart necklace 100 may provide output such as (person,desk, trash can) to indicate that within the viewing angle and from leftto right is a person, a desk and a trash can.

The smart necklace 100 may be able to determine preferences of the user.For example, the user may wish to know about certain items only, such aslarge, bulky items and people. This may also be set within the scope ofinformation setting. In this example, the smart necklace 100 may simplyinform the user of the person and the desk and not of the trash can.

If constant information is requested in block 1604, the processor 111may determine a desired viewing range in block 1610. The viewing rangemay be pre-determined or it may be selected by the user. The user maysay, for example, “what is to my sides,” or “what is directly ahead.” Invarious embodiments, the user may simply single click a button toindicate that he wishes to know what is to his sides or directly aheadand may double click to indicate the other selection.

In block 1612, the smart necklace 100 may detect and identify objectsand/or people within the desired viewing range. The desired viewingrange may include a pre-determined distance and angle, such as in block1606, or a user determined distance and angle.

In block 1614, information is output by the smart necklace 100 thatidentifies the objects and/or people via the interface array. Outputdetermined in block 1614 may be provided in the same manner as outputdetermined within block 1608.

In block 1616, it is determined whether the explorer mode has beencancelled. There are various ways in which the explorer mode may becancelled. The user may depress and release the corresponding button inorder to cancel the explorer mode. Similarly, the user may say “cancelexplorer mode.” In some embodiments, beginning a new mode will cancelthe explorer mode. In various embodiments, selecting a new mode willplace the explorer mode on hold until the new mode is complete.

In block 1616, if the explorer mode has been cancelled, then the method1600 ends. If explorer mode has not been cancelled, then the processreturns to block 1612 where objects are detected and determined withinthe desired viewing range. In various embodiments, the method 1600 maybe delayed a predetermined amount of time before the method 1600 returnsto block 1612. This allows all of the objects identified in the firstiteration of block 1612 to be identified and output to the user beforethe second iteration of block 1612. Additionally, a delay of apredetermined amount of time allows new objects to enter the area inwhich objects will be detected.

FIG. 17 illustrates two exemplary implementations of the method 1600. InFIG. 17, a user is at a location 1700. In a first implementation, theuser may request non-constant information to the sides of the user. Thesmart necklace 100 may detect image data on both sides of the userwithin a pre-determined distance 1713 and a pre-determined angle 1714.An area 1711 indicates the viewing area to the right of the smartnecklace 100 and an area 1712 indicates the viewing area on the left ofthe smart necklace 100. The smart necklace 100 may simply indicate thata table 1716 is present to the right of the user as the table 1716 isthe only object within the area 1711 or 1712.

In the second implementation, the user may request constant informationin front of the user. In this example, the smart necklace 100 may detectdata within a pre-determined distance 1702 and a pre-determined angle1703 of the smart necklace 100. This area relative to the smart necklace100 may be represented by the area 1701. Within area 1701 is a chair1704 and a desk 1706. The user may be walking as the smart necklace 100is providing this data to the user. By the time the smart necklace 100has indicate the presence of the chair 1704 and the desk 1706, the usermay be at location 1708. In location 1708, the smart necklace 100 mayagain detect objects within the area 1701. At location 1708, a desk 1710is detected within the area 1701. The smart necklace 100 may detect andidentify the desk 1710 and inform the user of the presence of the desk1710.

In various embodiments, while at location 1700, the smart necklace 100may inform the user that the chair 1704 is a first distance away fromthe smart necklace 100 and the desk 1706 is a second distance from thesmart necklace 100. In various embodiments, the smart necklace 100 maysimply list the chair 1704 and the desk 1706 to indicate that the chairis closer to the smart necklace 100 then the desk 1706.

FIG. 18 illustrates a method 1800 to be performed when the selected modeis the scan mode. The method 1800 may be performed by a smart device,such as smart necklace 100. The method 1800 may be performed while thesmart necklace 100 is in the scan mode and may be performed within thescan module 164 of the processor 111.

In block 1802, the processor 111 may determine settings of the smartnecklace 100 by accessing the memory 112. The settings may include ascope of information setting which may include a desired granularity ofinformation to be provided to the user.

In block 1804, the stereo camera 121 and/or the camera 122 may detectobjects and/or people within a pre-determined distance and angle. Thepre-determined distance and angle may be preset within the smartnecklace 100 and/or may be selected by a user, such as within thesettings. For example, the pre-determined distance may be 30 feet andthe pre-determined angle may be greater than 90 degrees.

The stereo camera 121 and/or the camera 122 may provide the detecteddata to the processor 111. The processor 111 may then compare the imagedata to data in the memory 112 in order to identify objects and/orpeople within the image data. In various embodiments, positioning datafrom the GPS 124 and/or the IMU 123 may be detected and provided to theprocessor 111. The identification of objects may be based on thepositioning data as well as the image data. For example, the processor111 may determine a location based on the positioning data. Theprocessor 111 may then compare the location to a memory to determinepotential matches for the detected objects, and then verify the matchbased on the image data.

In block 1806, the processor 111 may determine whether a single objector person is selected for identification. This determination may be madebased on user input. For example, the user may verbally say “who isthis,” or “what is this” to indicate that a single object or person hasbeen selected for identification. The user may also verbally say “whatis around me” to indicate that a single object or person is not selectedfor identification. In some embodiments, the user may depress andimmediately release a button to indicate a desire to identify a singleobject or person and may depress and hold the button to indicateotherwise. In some embodiments, a single depression and release of thebutton may indicate one preference and a double click of the button mayindicate the other.

In some embodiments, the processor 111 includes logic to determinewhether a single object or person has been selected for identificationbased on image data. For example, if the stereo camera 121 and/or thecamera 122 detect image data that includes a hand or hands of the userholding an object, the processor 111 may determine that the user wishesto identify the single object. Likewise, if the processor determinesthat a user is pointing at something or someone, the processor 111 maydetermine that the user wishes to identify that single object or person.Similarly, if a single large object or a person is detected immediatelyin front of the stereo camera 121 and/or camera 122, the processor maydetermine that the user wishes to identify the single object or person.

In block 1808, if a single object or person is selected foridentification, the smart necklace 100 may provide information to theuser via the interface array 130 that identifies the object or person.

In block 1810, if a single object or person is not selected foridentification, then the smart necklace 100 may output informationidentifying the objects and/or people detected within the pre-determineddistance and angle. The information provided by the smart necklace 100may be limited based on the settings of the smart necklace 100. Forexample, if the scope of information setting is large, more informationwill be provided to the user than if the scope of information setting issmall.

The information may be provided to the user in various manners. Forexample, the information may be provided to the user in a clock facemanner. In this embodiment, the smart necklace 100 may output the nameof the object and the clock face direction of the object, such as chairat your 10 o'clock. In various embodiments, the smart necklace 100 mayalso provide distance information to the objects and/or people. Invarious embodiments, the smart necklace 100 may list the objects and/orpeople in a particular order, such as left to right, right to left,closest to farthest, or farthest to closest.

In block 1812, it is determined whether navigation to a detected objectand/or person is requested. The processor 111 may determine whethernavigation is requested in multiple manners. In some embodiments, theuser may state that he or she wants directions to a detected objectand/or person. In some embodiments, the user may depress a button torequest navigation to a detected object or person. For example, as thesmart necklace 100 is outputting data identifying the objects and/orpeople, the user may press the scan button after hearing certain data toindicate that the user requests navigation to the previously outputtedobject or person. If navigation to the detected object or person is notrequested, then the method 1800 may end.

If navigation to a detected object or person has been requested, themethod may proceed to block 1814. In block 1814, the processor 111 maydetermine the location of the desired object or person on a map and/orrelative to the smart necklace 100.

In block 1816, the processor 111 may determine navigation instructionsfrom the smart necklace 100 to the desired object or person. Thenavigation instructions may be determined based on map data, image datadetected by the stereo camera 121 and/or camera 122, positioning datadetected by the IMU 123 and/or data detected by the GPS 124. In variousembodiments, image data detected by the stereo camera 121 and/or thecamera 122 may be positioning data as the processor 111 may determine apositioning of the smart necklace 100 based on a comparison of the imagedata to image data in memory. For example, a map in the memory 112 mayinclude image data associated with various locations. The processor 111may compare the detected image data to the stored image data todetermine a location of the smart necklace 100. The smart necklace 100may generate navigation instructions based on positioning on a mapand/or the location of the desired object or person relative to thesmart necklace 100.

In block 1818, the smart necklace 100 may provide navigationinstructions to the user via the interface array 130. The navigationinstructions may be provided in different manners as described above.The navigation instructions may be determined differently from above, asthe desired object or person is within a detectable distance from thesmart necklace 100. The navigation instructions may thus be more easilydetermined, as navigation instructions may be determined based on visualdata from the stereo camera 121 and/or the camera 122. In variousembodiments, the processor 111 may use map data and location data todetermine navigation instructions.

In block 1820, it is determined whether the smart necklace 100 is at thedesired object or person. If the smart necklace 100 is at the desiredobject or person, then the smart necklace 100 may alert the user to thisvia the interface array 130. If the device is not at the desired objector person, then the processor 111 may determine whether the user is onthe determined navigation path in block 1824. If the user is not on thedetermined navigation path, then the method may return to block 1816where new navigation instructions may be determined. If the user is onthe determined navigational path, then the process may return to block1818 where the navigation instructions may continue to be provided.

FIG. 19 illustrates an exemplary implementation of the method 1800. InFIG. 19, a user is wearing a smart device, such as the smart necklace100. While the user is at location 1900, the user may select the scanmode. In the example illustrated in FIG. 19, a single object or personis not selected for identification. Therefore, the smart necklace 100may detect and determine objects within a pre-determined distance 1902and a pre-determined angle 1903 of the smart necklace 100. An area 1901illustrates the area in which data will be detected and determined.Within the area 1901 is a table 1904, a table 1906, an information desk1908, two chairs 1910, a desk 1912, an L-shaped desk 1916, a trash can1918 and a staircase 1920. Depending on the scope of information settingof the smart necklace 100, the smart necklace 100 may provideidentification data to the user for all or some of these objects withinthe area 1901. The smart necklace 100 may provide this information usinga clock face type output, a direction type output, or an ordered output.

If the output is given in clock face format and the scope of informationsetting is medium, the smart necklace 100 may output, via the speakers132, “information desk at your 11 o'clock, square desk at your 12:30,L-shaped desk at your 1 o'clock and staircase at your 2 o'clock.” If theoutput is given in directional format and a low scope of informationsetting, the smart necklace 100 may output “information desk on the leftand staircase on the right.”

If the output is given in list format with a large scope of informationsetting, the smart necklace 100 may output “information desk, chair,chair, table, table, L-shaped table, table, trash can, stairs.” Thisindicates that the information desk 1908 is the first item on the leftside of the area 1901, that the two chairs 1910 are behind theinformation desk and the table 1912A is behind the chairs, that movingto the right is the table 1904, farther to the right is the L-shapeddesk 1916, farther to the right is the table 1906, then the trash can1918, then the staircase 1920. In some embodiments, the smart necklacemay also output the distance to each object within the area 1901.

FIG. 20 illustrates a method 2000 to be performed when the capture modeis selected. The method 2000 may be performed by a smart device, such asthe smart necklace 100. The method 2000 may be performed within thecapture module 166. While in the capture mode, the smart necklace 100may permanently label a person, location or object or may temporarilyremember a person, location or object.

In block 2002, it is determined whether the label sub-mode is selected.This may be determined by the processor 111 in a variety of manners. Forexample, a user may say “this is Mary,” “this is my work,” “this is acup,” or the like. In some embodiments, a single click of a button mayindicate the label sub-mode and a double click may indicate otherwise.In some embodiments, a depression and release of the button may indicatea label sub-mode and a longer depression may indicate otherwise.

In block 2004, if the label sub-mode is selected, the stereo camera 121and/or the camera 122 may detect image data.

In block 2006, the processor 111 may receive the image data anddetermine whether a single object or person is selected. Thisdetermination may be made based on the image data. For example, if theuser is pointing at a person or holding an object, the processor 111 maydetermine that the object or person is selected for labeling. Similarly,if a single object or person is in the field of view of the stereocamera 121 and/or the camera 122, the processor 111 may determine thatthat object or person has been selected for labeling. In someembodiments, the processor 111 may determine what is to be labeled basedon the user's verbal commands. For example, if the verbal commandincludes the name of an object that the processor 111 has identified,the processor 111 may know that the label is for that object. If thelabel includes a human name, the processor 111 may determine that ahuman is to be labeled. Otherwise, the processor 111 may determine thatthe current location is to be labeled. Additionally, if the user statesthe name of a location, such as “my workplace,” the processor 111 maydetermine that the location is selected for labeling.

In block 2008, the processor 111 may determine a label for the object orperson. The user may input the label via the input device 134 or byspeaking the label such that the smart necklace 100 detects the labelvia the microphone 131.

In block 2010, the processor 111 may store the image data associatedwith the object or person and the memory 112. The processor 111 may alsostore the label in the memory 112 and associate the label with the imagedata. In this way, image data associated with the object or person maybe easily recalled from the memory 112 because it is associated with thelabel.

In block 2012, the smart necklace 100 may inform the user via theinterface array 130 that the task is complete. For example, audio may beplayed over the speaker 132 such as “the label is stored.” In variousembodiments, a particular set of vibrations and/or a particular tone ortones may be provided to the user to inform the user that the task iscomplete.

If it is determined in block 2006 that a single object or person is notselected, the smart necklace 100 may determine the current location ofthe smart necklace 100 in block 2014. The location may be determinedusing data from the sensor array 120, such as visual data from thestereo camera 121 and/or the camera 122, the IMU 123, the GPS 124 and/orthe sensor 125.

In block 2016, the processor 111 may determine a label for the currentlocation. The label may be provided by the user via the input device 134and/or the microphone 131.

In block 2018, the processor 111 may store the current position and thelabel in the memory 112. The processor 111 may also associate thelocation with the label such that the location information may beretrieved from the memory 112 using the label. In some embodiments, thelocation may be stored on a map.

In block 2020, the smart necklace 100 may inform the user that the taskis complete in the same manner as in block 2012.

If it is determined in block 2002 that the label sub-mode is notselected, then the processor 111 may determine the current locationusing data from the sensor array 120 in block 2022. The processor 111may also store the current location in the memory 112 such that it canbe retrieved at a later time. The current location may be stored in mapdata of the memory 112. In various embodiments, the present location isdetermined based on image data alone or a combination of image data anddata from the IMU 123.

In various embodiments and as the smart necklace 100 changes location,data may be continuously or periodically stored in the memory 112 as thesmart necklace 100 changes location. This data may be retrieved at alater time and used to provide directions from a later location of thesmart necklace 100 to the stored location of the smart necklace 100.

In block 2024, the processor 111 determines whether the smart necklace100 has been instructed to return to the location stored in memory. Insome embodiments, a user may depress the button to instruct the smartnecklace 100 to return to the position. In some embodiments, the usermay say a command such as “take me back” to indicate a desire to returnto the previous location.

In block 2026, if the smart necklace 100 has been instructed to returnto the previous location, the processor 111 may determine the currentlocation of the smart necklace 100 using data provided from the sensorarray 120. The processor 111 may also retrieve the informationassociated with the stored location from the memory 112.

In block 2028, the processor 111 may compare the current location to thestored location and determine navigation instructions from the currentlocation to the stored location. The processor may use a map and/or datafrom the sensor array 120 to determine the navigation instructions. Invarious embodiments, the processor 111 may determine the navigationinstructions based on a single type of data, such as positioning data orimage data. For example, the processor 111 may receive image data fromthe start of block 2022. This image data may be stored such that theprocessor may later compare newly detected image data to the storedimage data in order to determine directions to the stored location.

In block 2030, the smart necklace 100 may provide the navigationinstructions via the interface array 130. The instructions may beprovided using clock face descriptions, directional instructions, tonesfrom the speakers 132 and/or vibrations from the vibration unit 133.

In block 2032, the processor 111 may compare the current location of thesmart necklace 100 to the previous location stored in memory. If theprocessor 111 determines that the smart necklace 100 is at the desiredlocation, then the smart necklace 100 may alert the user via theinterface array 130.

In block 2036, the processor 111 may determine whether the user is onthe determined navigation path. If the user is on the determinednavigation path, the process may return to block 2030 where navigationinstructions will continue to be provided to the user. If the user isnot on the determined navigation path, the process may return to block2028 where navigation instructions will again be determined from thecurrent location of the smart necklace 100 to the stored location.

FIG. 21 illustrates an exemplary implementation of the method 1400 andtwo exemplary implementations of the method 2000. In FIG. 21, a user ofa smart device such as the smart necklace 100 may initially be sittingin a chair 2102A. The user may desire to go to the women's restroom2106. The user may first select the capture mode to temporarily storethe location 2100 which is the location of the chair 2102A. The user mayindicate a temporary location save by single clicking the capturebutton. The user may then select the find mode by depressing the findbutton and informing the smart necklace 100 that she wishes to go to thewomen's restroom 2106.

The processor 111 may then determine the location of the women'srestroom 2106 relative to the smart necklace 100 at location 2100. Theprocessor 111 may then determine navigation instructions from thelocation 2100 to the women's restroom 2106. The navigation route isindicated by paths 2110, 2112, 2114, and 2118.

As the user is walking along path 2114, the user may hear the voice of aperson 2108. The user may then stop at a position 2116 near the person2108 in order to talk to the person 2108. The user may desire that thesmart necklace 100 label the person 2108. The user may then select thecapture mode again and indicate that a label sub-mode is selected bydepressing and holding the capture button. The processor 111 may thendetermine that the person 2108 is to be labeled. The user may say “thisis Tom.” The processor may then store image data associated with theperson 2108 and the label Tom and associate the label with the imagedata. The user may then continue walking along path 2114 to indicate tothe smart necklace 100 that the user wishes to continue the navigationinstructions. In some embodiments, the user may depress the find buttononce to pause the navigation and depress the find button again tocontinue the navigation.

When the user is ready to return to the location 2100, the user mayagain select the capture mode. For example, the user may depress thecapture button to indicate a desire to return to the stored location. Insome embodiments, the user may speak a command, such as “return toprevious location,” to indicate a desire to return to the storedlocation. The smart necklace may then retrieve the stored location anddetermine navigation instructions from the current location of the smartnecklace 100 (the women's restroom 2106) to the stored location 2100.The smart necklace 100 may then provide navigation instructions to theuser such that the user can return to the location 2100 by following thedirections.

FIG. 22 illustrates a graphical user interface (GUI) 2200 displayed onthe display 135 that illustrates various settings of a smart device suchas the smart necklace 100. The values for the settings are exemplaryonly, and one skilled in the art will realize that the selectable valuesmay be more or less inclusive than shown in FIG. 22. GUI 2200 is meantto be exemplary only and one skilled in the art will understand that theGUI 2200 may have a totally different look. In various embodiments, thesettings are not displayed on a display and are simply stored in thememory 112. Various embodiments of the settings may include all or someof these setting, as well as additional settings not illustrated in FIG.22. Additionally, some or all of these settings may or may not beselectable by a user of the smart necklace 100.

The display 135 is connected to the processor 111 such that theprocessor 111 may provide the data to be displayed. The current settingsof the smart necklace 100 may be displayed in the GUI 2200. A user maychange some or all of these settings using the input device 134 and/orvoice commands. When a setting is changed, the new setting may be storedin the memory 112 and implemented by the processor 111. In someembodiments, the processor 111 may learn user preferences over a periodof time. For example, if a user always selects silent mode for aparticular location, the smart necklace 100 may automatically place thesmart necklace 100 in silent mode in response to arriving at theparticular location.

The GUI 2200 may include an audio and vibrate setting 2202. The audioand vibrate setting may indicate that the user wishes to receive bothaudio and haptic feedback. For example, during navigation instructions,the navigation instructions may be provided using a combination of audioand haptic cues. Similarly, audio and haptic feedback may be provided tothe user in any mode of the smart necklace 100.

The GUI 2200 also includes a silent mode setting 2204. When the smartnecklace 100 is in the silent mode, only haptic feedback will beprovided to the user. This may be preferred if the user is in a largegroup setting and does not want audio feedback from the smart necklace100 to disturb others.

The GUI 2200 also includes a granularity setting 2206. The granularitysetting 2206 can be used to set the desired scope of information. Whenthe granularity setting 2206 is set to a higher value, such as five,more detail may be provided to the user about more objects. When thegranularity setting 2206 is low, such as 1, fewer details may beprovided about fewer objects.

Also included is an information refresh rate setting 2208. Theinformation refresh rate setting 2208 may indicate how often the userdesires to receive new information. For example, if the user isreceiving navigation instructions and the refresh rate is high,instructions may be provided to the user at a higher rate, such as every10 seconds instead of every 30 seconds. The information refresh rate mayalso be used in the explorer mode. For example, if the informationrefresh rate is low while the user is in a constant information scanmode, the smart necklace 100 may only provide new object recognitioninformation to the user at a slower rate than if the refresh rate werehigher.

The GUI 2200 also includes a volume setting 2210. The volume setting2210 may indicate a volume at which audio data will be provided.

Another available setting is a vibration intensity setting 2212. Thevibration intensity setting 2212 may indicate a desired intensity ofvibrations to be provided via the vibration unit 133.

Also included is a speed of speech setting 2214. Some users may desireto receive speech at a faster rate than other users. For example, blindusers place more importance upon sounds than non-blind users, so a blinduser may be able to hear and interpret speech faster than other users.

The GUI 2200 also includes a tone of speech setting 2210. Some users maybe more comfortable with certain tones of voice while other users aremore comfortable with other tones of voice. The tone of speech setting2216 allows the user to change the tone of the speech to better suit hisdesires.

The GUI 2200 also includes a type of verbal information setting 2218.The type of verbal information setting 2218 may include a clock facetype, a left to right type, a right to left type and a directional type.

The GUI 2200 may also include a type of navigation instructions setting2230. The type of navigation instructions setting 2230 may includevibration type, tones type, clock face type and directional. In someembodiments, more than one type of navigation instructions may beselected, such as vibration and directional. If the vibration type isselected, navigation instructions may be provided via vibrations fromthe vibration unit 133. If the tones type is selected, the smartnecklace 100 may provide navigation instructions using certain tones orcombination of tones. If the clock face type is selected, the smartnecklace 100 may provide navigation instructions using clock facedirections, such as (proceed 10 feet and turn to your 2 o'clock, proceed10 feet and turn to your 3 o'clock.) If the directional type isselected, the smart necklace may provide directional navigationinstructions, such as (walk 20 feet and turn right, walk 10 feet andturn left.)

Also included in the settings is a vibration/tone navigation setting2232. The vibration/tone navigation setting 2232 may include a normaltype and a bumper type. If the normal type is selected, a vibrationand/or tone may be played on the side to which the user should turn. Ifthe bumper type is selected, a vibration and/or tone may be played onthe side to which the user should turn away from.

The GUI 2200 may also include a provide distance information setting2234. If yes is selected, the smart necklace 100 may provide distanceinformation to the user. This distance information may be provided inany mode of the smart necklace 100.

Exemplary embodiments of the methods/systems have been disclosed in anillustrative style. Accordingly, the terminology employed throughoutshould be read in a non-limiting manner. Although minor modifications tothe teachings herein will occur to those well versed in the art, itshall be understood that what is intended to be circumscribed within thescope of the patent warranted hereon are all such embodiments thatreasonably fall within the scope of the advancement to the art herebycontributed, and that that scope shall not be restricted, except inlight of the appended claims and their equivalents.

What is claimed is:
 1. A method for providing directions to a blind userof a smart device, the directions being from an initial location of thesmart device to a location of a desired object or a desired location,the method comprising: detecting, by at least two sensors and inresponse to a selection of a find mode of the smart device, image datacorresponding to a surrounding environment of the smart device andpositioning data corresponding to a positioning of the smart device;receiving, by an input device, the desired object or the desiredlocation; determining, by a processor, the initial location of the smartdevice based on the image data, the positioning data and map data storedin a memory of the smart device; and providing, by the output device,the directions to the desired object based on the initial location ofthe smart device and the map data.
 2. The method of claim 1 furthercomprising generating, by the processor, the map data based on the imagedata and the positioning data.
 3. The method of claim 1 furthercomprising: determining, by the processor, that the desired object orthe desired location is not stored in the memory; requesting, by theprocessor, additional data from the blind user identifying the desiredobject or the desired location; and determining, by the processor, thedirections to the desired object based on the initial location of thesmart device, the map data and the additional data identifying thedesired object or the desired location.
 4. The method of claim 1 whereinthe output device includes a first vibration unit positioned on a firstside of the smart device and a second vibration unit positioned on asecond side of the smart device and the directions include vibrationalpatterns output from the first vibration unit and the second vibrationunit.
 5. The method of claim 1 wherein the output device includes aspeaker and the directions are provided by the speaker in speech output.6. The method of claim 1 further comprising determining, by theprocessor, that an obstacle exists in a current path of the smartnecklace based on the image data and wherein providing the directions tothe desired object includes providing directions that cause the blinduser to avoid the obstacle.
 7. A method for describing at least oneobject or person within a predetermined distance and angle of a smartdevice to a blind user of the smart device, the method comprising:storing, in a memory, stored image data associated with a plurality ofstored objects and people and a plurality of identifiers such that eachidentifier is associated with a stored object or person; detecting, by acamera and in response to a selection of an explore mode or a scan modeof the smart device, detected image data corresponding to the at leastone object or person; determining, by a processor, at least a firstidentifier from the plurality of identifiers corresponding to the atleast one object or person within the predetermined distance and angleof the camera based on the detected image data and the stored imagedata; and outputting, via a speaker, the at least first identifier. 8.The method of claim 7 wherein the predetermined distance and angle ofthe camera is less than a field of view of the camera.
 9. The method ofclaim 7 further comprising: determining, by the processor and inresponse to the selection of the explore mode, if the explore mode hasbeen canceled; detecting, by the camera and in response to the exploremode not being canceled, a second image data of at least another objector person after a predetermined period of time; determining by theprocessor, at least a second identifier from the plurality ofidentifiers corresponding to the at least another object or personwithin the predetermined distance and angle of the camera based on thesecond image data and the stored image data; and outputting, via thespeaker, the at least second identifier.
 10. The method of claim 7,further comprising determining, by the processor, whether a specificobject or person is selected for identification based on the detectedimage data and wherein the first word or phrase includes a specific wordor phrase associated with the specific object or person.
 11. The methodof claim 7, wherein: the at least one object or person includes a firstobject or person, a second object or person and a third object orperson; the at least first identifier includes a first identifiercorresponding to the first object or person, a second identifiercorresponding to the second object or person and a third identifiercorresponding to the third object or person; and outputting the at leastfirst identifier includes outputting the first identifier, the secondidentifier and the third identifier based on a directional order of thefirst object or person, the second object or person and the third objector person.
 12. The method of claim 7, wherein the camera includes a pairof stereo cameras, the detected image data includes depth informationand outputting the at least first identifier further includes outputtingthe depth information.
 13. The method of claim 7 further comprising:receiving, at the processor, a request for directions to the at leastone object or person; determining, by the processor, directions from acurrent location of the smart device to a location of the at least oneobject or person; and outputting, via the speaker, the directions. 14.The method of claim 7, wherein the predetermined distance and thepredetermined angle are different based on whether the explore mode orthe scan mode were selected.
 15. The method of claim 7 furthercomprising determining, by the processor, a granularity setting of thesmart device wherein the image data corresponds to a first object and asecond object, determining at least a first identifier includesdetermining a first identifier corresponding to the first object and asecond identifier corresponding to the second object and outputting theat least first identifier includes outputting the first identifier onlybased on the granularity setting.
 16. A method for storing a firstlocation of a smart device and providing directions to a blind user froma second location of the smart device to the first location of the smartdevice, the method comprising: detecting, by a positioning sensor, afirst positioning data corresponding to a first location of the smartdevice and a second positioning data corresponding to a second locationof the smart device; storing, in a memory and in response to a firstselection of the capture mode of the smart device, the first positioningdata on a map; determining, by a processor and in response to a secondselection of a capture mode of the smart device, directions to the firstlocation from the second location based on the first positioning dataand the second positioning data; and providing, by an output device, thedirections.
 17. The method of claim 16 further comprising: detecting, bya camera, image data corresponding to an object or a person;determining, by the processor, that the object is selected for labeling;receiving, by an input device, a label for the object or person;storing, in the memory, the image data and the label; and associating,in the memory, the image data with the label.
 18. The method of claim 16further comprising detecting, by a camera and in response to the secondselection of the capture mode, image data corresponding to anenvironment of the user and wherein determining the directions includesdetermining the directions based on the image data.
 19. The method ofclaim 18 wherein determining the directions includes: determining aroute between the first location and the second location; determining anobstacle along the route based on the image data; and determining analternate route based on the obstacle.
 20. The method of claim 16,further comprising: determining, by the processor, that the firstlocation is selected for labeling; receiving, at the processor, a labelfor the first location; storing, in the memory, location data associatedwith the first location and the label; and associating, in the memory,the location data with the label.